[question] Understanding the cloudpath upload on close mechanism

[fafnirZ]

Hi I have a general question on how the uploading to cloudpath works.

My current understanding it this:
When you path.open("w") as f where path is a cloudpath, it just streams the data into a local temporary file somewhere on your local FS. And within the cloudpath.open functions it wraps the buffer.close method with the custom implementation which uploads the temporaryfile to the cloud when the stream is closed.

maybe this is a general python question, but how does it enforce that the .close() function is actually called in the event of the program terminating unexpectedly?

Is it the user's responsibility to either wrap it in a context manager in a with ... block to ensure that the _patched_close_upload is properly called in an event of an program exit?

In a nutshell, how do I ensure that the file actually is written to cloud under all circumstances?

Forgive me if its a silly question, any pointers are greatly appreciated :)

[pjbull]

@fafnirZ You have the right understanding. In order to get the upload to happen, an end user must either (1) use a context manager and a with statement, which is the most idiomatic, or (2) track the buffer and call .close (probably in a finally block to make sure it executes).

That said, there are possible but unlikely catastrophic failures where Python is killed and does not shutdown properly and in these scenarios the file will not be uploaded.

[fafnirZ]

Hi @pjbull, Thanks for your response, appreciate it!
In terms of python getting killed, are the only 2 scenarios when a SIGKILL signal has been has been issued or when the program has been evicted by OOMKiller?

[pjbull]

Yeah, those are the two most likely scenarios at the OS level (i.e., not an operating system or hardware failure}.

[fafnirZ]

sounds good, thanks for clarifying!

[jayqi]

@pjbull I'm wondering if we should call this out somewhere in the documentation since it's critical to the successful use of cloudpathlib, and not all users might default to following best practices around closing file handles.

Something like:

Important

When writing to the cloud, cloudpathlib uploads the file when the file handle is closed. Be sure to use CloudPath(...).open("w") as a context manager using the with statement, or call .close() (consider a finally to upload even in case of exceptions).

in the basic usage section of the README maybe.

[pjbull]

Added to our docs tweaks issue FWIW #133