Let's use streams

[dgrammatiko]

Possible solution instead of implementing pagination

Stream what?

So right now whenever a call from the from end is done to get the json for the items (files) in a directory the process is like: server reads the files -> creates the appropriate json -> sents back the json. Using stream we can have this process asynchronous, meaning that instead of waiting to get the complete json we could start decoding the stream and build the json (the front end store data) as the data is streamed from the backend, almost realtime. Now if we combine this with some clever lasyloading (intersection observer) then we will end up with the exact behaviour that google has in their search images page! WIN

What we need?

Three parts need to come together here:

• Sever side something like https://github.com/violet-php/streaming-json-encoder to stream the json data

• Client side: a chunk decoder (here is a how to: https://streamdata.io/blog/streamdataio-vue-js-sample-realtime/) and maybe https://github.com/developit/stockroom to get the store in another thread (through a worker)

• Also client side: Lasyloading of thumbs using intersection observer https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API
  and a polyfill for IE11: https://github.com/jeremenichelli/intersection-observer-polyfill

[schnuti]

Don't you first need some sized images? i.e. thumbs. As I understand it, the search engines do not send the original sizes. I have no clue how the browser and the client stores the data but I have a feeling that there is a limit somewhere.

[dgrammatiko]

We already have the functionality for the thumbs (or at least that's my impression)

[schnuti]

And I have another impression see #475 and my own #255 and after checking the background images size.

800px_cradle_mountain_seen_from_barn_bluff.jpg is in real on disc a 500x375 image and that's the size of the background image.

[laoneo]

Sounds a nice approach. From a quick glance, not sure what it has for influence to the API. Will it still work as before and the streams are an addition on the top or will the API change by itself?

[dgrammatiko]

I guess the getFiles() part needs some refactoring to :

$iterator = function () {
    foreach (new DirectoryIterator(__DIR__) as $file) {
        yield $file->getFilename();
    }
};

$encoder = (new \Violet\StreamingJsonEncoder\BufferJsonEncoder($iterator))->setOptions(JSON_PRETTY_PRINT);

$stream = new \Violet\StreamingJsonEncoder\JsonStream($encoder);
}

[laoneo]

Can this lead to conflicts with #175 then?

[dgrammatiko]

I don't see any problem, eg if you search for someFilename the back end will stream something (with some value in the json if the file exists or nothing if the file doesn't exist) so the end result result will be the same json the backend could throw if it wasn't streamed. Maybe the implementation of the search might be slightly different in the front end (code wise) but nothing too radical or too special. Unless I'm missing something crucial here