Out of memory while downloading large files

[pall-valmundsson]

While trying to download a large .zip file (1.3GB) in a test VM with 512MB of RAM the only output I got from the puppet run was Killed. I increased the VM RAM incrementally and got Error: Could not run: failed to allocate memory until I had increased the RAM to 2GB, i.e. the file could be stored in RAM as a whole until flushed to disk.

Is this a known unfixable issue? Or an issue with the provider implementation? Or is there some workaround that can be applied?

[nibalizer]

Are you pulling the file over http or ftp?

[pall-valmundsson]

Ah, sorry I left that out. This is a HTTP Basic Auth pull.

[nibalizer]

Looking a bit at the code, this looks like an issue in the faraday library. However I dont see an issue on the faraday bug tracker for it. I'll try to find some time to bang on it today.

[pall-valmundsson]

Thanks for looking into this. Other potentially important information.
Puppet 3.7.5
CentOS 6
and therefore Ruby 1.8.7

[nibalizer]

I am able to reproduce the problem at least.

[nanliu]

In this case might as well swap out faraday with net/http and eliminate a dependency. I'll take a crack at it over next few days.

[nibalizer]

@nanliu do you think its in faraday? Is the typical pattern in puppet to hold in ram before file flushing? I'm just not experienced enough to know.

[nanliu]

I thought this was one of things I verified early on, but clearly missed it when evaluating faraday. Per their roadmap: https://github.com/lostisland/faraday/wiki/Roadmap and open PR: lostisland/faraday#156, I don't think we are able to do streaming with faraday. net/http will require some code to handle http errorcodes and redirect, but will support streaming.

[jyaworski]

Since it looks like the Faraday PR is stalled, should we move to net/http or similar?

[nanliu]

Yes, we should switch to net/http as default provider.