Temporary files are an inefficient use of cloud storage

[dralley]

Version
All

Describe the bug

Pulp uses temporary files and puts them on the same storage backend as everything else (AFAIK) - storage backends like S3 or Azure require you to pay for usage, so using those services for short lived and often small files is not an efficient use of resources.

[ggainey]

See also https://discourse.pulpproject.org/t/temporary-storage-on-s3/828/6

[ggainey]

Discussion in Matrix: https://matrix.to/#/!OkZKBrgwXfnkOdFreH:matrix.org/$z1Mg8nM2MdMxFKCteYIf2PPm2tkRWlrR1ZGF5lZ3HdA?via=libera.chat&via=matrix.org&via=x9c4.spdns.org

tl;dr: it's complicated and needs a lot more thought.

[pedro-psb]

The Pulp has discussed that on the pulpcore meeting today:

I've presented an initial idea to start the discussion:

See diagram


Some of key takes from the discussion were:

• Allow API and Workers on same host is not a good idea:
  • probably fixing SFTPStorage (from django-storages, which is not thread-safe) would be easier than support this
• Is adding the extra temporary_storage to be used be PulpTemporaryFile worth it?
  • Since we can't use shared FS in the cloud, we would end up using S3 for this anyway. Any advantage on that?
• The S3 transfers shouldn't cost anything on cloud deployment in the same region. If so, this is more a performance issue.
  • TODO: find reference
• Can we make object storage traffic faster (leveraging configs)? Like for parallel upload/download.
  • We should learn more about cloud storage.
  • https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig
• Tradeoffs are too abstract for cloud storage transfers. Can we get some real data?
  • setup some experiment to measure file transfers on cloud setups (e.g, instance->s3 vs EBS)