fs.put_file flakily uploading 0 bytes

[brandon-leapyear]

We're running into a flaky issue where using fsspec to upload data to WebHDFS is uploading 0 bytes. Printing the contents of the file before and after fs.put_file() confirms that the file on disk is non-empty. I have also found that this line is returning an empty string on the first (and only) iteration of the loop:

filesystem_spec/fsspec/spec.py

Line 813 in b45ae3e

data = f1.read(self.blocksize)

This is happening in CI (Circle CI), so resource/disk constraints is not out of the question, but it's very odd that I can read the file before and after. Let me know what other information would be helpful to debug and I can try to get those to you.

[martindurant]

Whilst read() is not required to return as many bytes as requested, I've never seen it return none! I suppose the condition on the loop could be while f2.tell() < size where size was implicitly found anyway on L806.

[brandon-leapyear]

✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨

---

ah yeah, that would probably be a good improvement. I don't have enough details to determine if that would fix the problem or not (and since it's flaky, it's hard to determine if it worked after making the change)

There was another odd failure where after fsspec.put_file was called, I redownloaded the file from HDFS and it showed a file with ~2000 zero bytes and then ~150 bytes from the middle of the actual file (i.e. the original file was ABCDEF, the uploaded file was 000D). That failure only happened once, though, so we could treat that failure as a cosmic ray.

[martindurant]

I am inclined to believe in the cosmic ray.
You might find fewer weird problems - or at least clearer errors - with more highly secured endpoints and avoiding proxies. That probably doesn't apply to your CI, though.

[brandon-leapyear]

✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨

---

Update: still not seeing the 0 bytes failure. I am noticing something weird where f.seek(0, 2) is returning the wrong length, which happens in both my debugging code and in in put_file:

filesystem_spec/fsspec/spec.py

Lines 805 to 806 in 704ed91

	with open(lpath, "rb") as f1:
	size = f1.seek(0, 2)

Posted in Reddit here: https://www.reddit.com/r/learnpython/comments/rxmq43/help_with_python_file_flakily_not_returning_full/

This might be the key part of the flake that we're seeing.

[martindurant]

Thanks for posting that question, perhaps someone knows. You might also try StackOverflow.

[brandon-leapyear]

✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨

---

Update: I forgot that we're running tests in parallel, so one thread might be overwriting a file that another thread is trying to upload. We did write a safe_put() wrapper to protect race conditions with multiple threads uploading the same file, but I forgot to protect the initial write from race conditions.