Successful `import-url` / `push` resulted in truncated cache file

[ryan-williams]

Bug Report

Description

Here is a GitHub Action that ran:

dvc import-url s3://tripdata/202505-citibike-tripdata.zip s3/tripdata/202505-citibike-tripdata.zip
# Importing 's3://tripdata/202505-citibike-tripdata.zip' -> 's3/tripdata/202505-citibike-tripdata.zip'
dvc import-url s3://tripdata/JC-202505-citibike-tripdata.csv.zip s3/tripdata/JC-202505-citibike-tripdata.csv.zip
# Importing 's3://tripdata/JC-202505-citibike-tripdata.csv.zip' -> 's3/tripdata/JC-202505-citibike-tripdata.csv.zip'
dvc push
# 2 files pushed

However, the first imported file (s3/tripdata/202505-citibike-tripdata.zip) ended up truncated, in my S3 remote cache.

I backed up the truncated blob with a .bad suffix, and then manually fixed the blob in the cache (with aws s3 cp, dvc add, dvc push):

aws s3 ls s3://ctbk/.dvc/files/md5/9e/880ca091cc946d563ea4b115ec443e
# 2025-06-06 19:44:58  844607858 880ca091cc946d563ea4b115ec443e
# 2025-06-06 19:39:50  838860800 880ca091cc946d563ea4b115ec443e.bad

Verifying that 9e/880ca091cc946d563ea4b115ec443e.bad is a prefix of the full blob:

aws s3 cp s3://ctbk/.dvc/files/md5/9e/880ca091cc946d563ea4b115ec443e.bad - | md5sum
# ef7b7328a690dfdc9858c2da4cad9f41  -
bad_size="$(aws s3 ls s3://ctbk/.dvc/files/md5/9e/880ca091cc946d563ea4b115ec443e.bad | awk '{print $3}')"; echo $bad_size
# 838860800
aws s3 cp s3://ctbk/.dvc/files/md5/9e/880ca091cc946d563ea4b115ec443e - 2>/dev/null | head -c "$bad_size" | md5sum
# ef7b7328a690dfdc9858c2da4cad9f41  -

Reproduce

I'm guessing it was a transient issue in my GHA run. I haven't tried to reproduce it.

I'm not sure which one failed here:

• It could be that import-url failed, dvc push happily pushed the truncated blob
• Or import-url may have been fine, but push silently failed to complete.

Expected

If import-url or push fails to import or push a full file, the command should exit non-zero, and some errors should be logged.

Environment information

You can see everything in the the GHA:

• ubuntu-latest
• pip install output shows dvc-3.59.2 and etc.

[skshetry]

I'm not sure which one failed here:

• It could be that import-url failed, dvc push happily pushed the truncated blob

• Or import-url may have been fine, but push silently failed to complete.

Is this the commit? hudcostreets/ctbk.dev@5f16901

The size matches in the .dvc file for outs and deps, so it may be the latter.

When there is only a single file to push, dvc uses max_concurrency to do a multipart upload, so it could be that something happened there.

[skshetry]

The default chunksize is $50 \cdot 2^{20}$. The Linux VM in GHA has 4 cores, and DVC uses a max_concurrency of $4 \cdot \text{cpu_count()} = 16$.

$16 \cdot 50 \cdot 2^{20}$ is exactly 838860800. So, looks like s3fs did not upload the last chunk left (which is smaller than the default chunksize).

[skshetry]

Looks like this has been fixed in s3fs==2024.12.0, while you are using 2024.9.0.

• Fix a bug in the _upload_file_part_concurrent method fsspec/s3fs#910

Please update, and if that happens again, please feel free to reopen this.

[ryan-williams]

Thank you! I wonder if it makes sense to add some exclusions for known-buggy s3fs versions in dvc-s3 (e.g. like laminlabs/lamindb-setup#911).

[skshetry]

Thank you! I wonder if it makes sense to add some exclusions for known-buggy s3fs versions in dvc-s3 (e.g. like laminlabs/lamindb-setup#911).

Contributions are welcome. This bug has been there since 2024.3.0 (fsspec/s3fs@caf15c8) so I think it'd be better to simply raise the minimum requirement.