[general] Question: hashing order of operations

[noamross]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Quick question: Does targets hash target objects in-memory before serialization to files, or does it hash the files of objects after serialization?

We recently discovered a source of non-determinism for parquet files (apache/arrow#40361), which helped us understand why we got different target hashes for parquet files written on different platforms. In this case, we were using format = "file", writing to disk inside our functions. I assume in this case serialization of the file on-disk is performed. We were wondering if instead we used a target "parquet" format or custom format of an object we would expect to have identical hashes, which would be the case if hashing occurred in-memory before serialization.

@emmamendelsohn

[wlandau]

For in-memory non-target global objects, the object itself is hashed in memory. For functions, the function body is deparsed (to return a character string) and then the character string is hashed. For actual targets, the file itself is hashed, not the object in memory.

If you have a custom deterministic way of saving parquet files, then tar_format() might help. But if it's a standard argument to arrow, you might be able to find it in tar_resources_parquet(). And if it's an easy enough fix, a PR in targets to update https://github.com/ropensci/targets/blob/main/R/class_parquet.R would be great, particularly before the fix of #1244 makes it into the next release.

[noamross]

Thanks! Unfortunately we can't control the non-determinism, and if the on-disk targets are hashed they will be different. This prevents me from going down another rabbit hole, though. This should eventually be fixed on the arrow side (if I can fight my way through enough C++ for this PR 😄 apache/arrow#40392)

[noamross]

Oh, a follow-up, which is related to #1244, as well: does any hashing or serialization occur in crew workers? I understand marshalling and unmarshalling serializes for transfer from the worker to the main node. But in any cases does the worker serialize/hash, for instance when sending to cloud storage? If, in that case, one might have to think if there are platform-specific outcomes in serialization.

[wlandau]

mirai (the backend of crew) serialized data when it sends it to workers. But you can avoid all that, and marshalling/unmarshalling arrow data, by making the workers save/load the data using retrieval = "worker" and storage = "worker" in tar_target().