Wouldn’t be great to process a large HTTP response, in chunks, with Elixir streams?
HTTPStream.get("https://.../large_image.tiff") |> StreamGzip.gzip |> Stream.into(File.stream!("image.tiff.gz")) |> Stream.run
Or process a huge remote text files line by line, on the fly?
"https://.../numbers.txt" |> HTTPStream.get() |> HTTPStream.lines() |> Stream.map(...) |> Enum.sum()
In these two articles we see how to build an Elixir Stream around a HTTPoison async response so we can take advantage of streams lazyness and composability.
- Part 1 - Elixir Stream to process large HTTP responses on the fly
- Part 2 - Elixir Stream and large HTTP responses: processing text
This code is just meant as an experiment and it's not intended to be used in production!