[WIP] Reduce memory use with fetch and slurp

[sivel]

SUMMARY

This PR attempts a few things:

1. Use context managers to open files in a few locations
2. Where possible, don't keep multiple copies of the file in memory
3. Don't assume because we use become that a non-become user (scp/sftp/dd) cannot access the file. As such, don't run stat with become, so we know whether slurp is actually required.
4. Have slurp return the checksum of the data, so we don't have to use the data on the controller to calculate it.

ISSUE TYPE

• Bugfix Pull Request

COMPONENT NAME

lib/ansible/module_utils/basic.py
lib/ansible/modules/net_tools/basics/slurp.py
lib/ansible/plugins/action/__init__.py
lib/ansible/plugins/action/fetch.py

ADDITIONAL INFORMATION

[sivel]

We will still end up with 2x here. I'm interested in investigating whether we can write this to a real file, and not memory, and then have our JSON encoder be able to stream from a file handle. I attempted this before, but need to look again.

[ansibot]

The test ansible-test sanity --test pylint [explain] failed with 1 error:

lib/ansible/modules/net_tools/basics/slurp.py:91:14: undefined-variable: Undefined variable 'to_bytes'

The test ansible-test sanity --test pep8 [explain] failed with 3 errors:

lib/ansible/modules/net_tools/basics/slurp.py:76:26: E226: missing whitespace around arithmetic operator
lib/ansible/modules/net_tools/basics/slurp.py:76:30: E226: missing whitespace around arithmetic operator
lib/ansible/plugins/action/fetch.py:39:1: E303: too many blank lines (3)

click here for bot help

[sivel]

Ultimately this PR can only easily aim to improve memory on the controller. To reduce memory on the target further would require JSON gymnastics, but I question the benefit there too.

The big change on the controller, is instead of just doing f.write(base64.b64decode(data)) we base64 decode smaller chunks and stream to a file. As well as calculating the checksum via slurp on the target, so we don't have to use memory while hashing the base64 decoded content.

[jborean93]

I know @bcoca was looking to see if it's possible to buffer the data back in chunks and just rebuild the file on the controller only. The trouble is with the split done the Windows slurp.ps1 is now in a collection so any changes to the interface needs to be done in a backwards compatible manner.

[sivel]

any changes to the interface needs to be done in a backwards compatible manner

That should already be done. It should work with slurp.ps1 currently without requiring any changes. I'm still just playing though. I might throw this away like I have every time in the past.

Streaming the data back in chunks via "update_json" wouldn't necessarily address it either. I think in the end, we'll have to pay the penalty either on the controller or the target. I don't know which is preferable currently.

[jborean93]

It's mostly around your point of

Have slurp return the checksum of the data, so we don't have to use the data on the controller to calculate it.

I see you've made that an optional check but I just wanted to make sure that you were aware of it, which it seems like you are.