update: import-url updates will get re-downloaded even if nothing changes

[dberenbaum]

Bug Report

Description

This is a regression starting with #8024. After dvc import-url, dvc update will always re-download data even if it hasn't changed.

Reproduce

For any external data, do dvc import-url followed by dvc update.

Expected

Before this change, dvc update would return fast with output like 'wiki.tar.gz.dvc' didn't change, skipping.

[dtrifiro]

Where is the data being imported from?

[sheluchin]

I had this issue come up when importing release assets from GitHub using the latest form of the URL, like https://github.com/USER/PROJECT/releases/latest/download/package.zip. It has an etag, but the etag doesn't show up in the corresponding .dvc file - only an md5 and checksum (which actually holds the etag value). When I do dvc update subsequently, it downloads the whole file again. My expectation is that there should not be a download since the etag is the same.

[dberenbaum]

@dtrifiro It looks like it's caused by:

https://github.com/iterative/dvc/blob/f386f7c082717f986e2bbd8b53e05eb87616ccb2/dvc/stage/imports.py#L30-L31

This causes the stage to always look changed because the outs are removed before checking for changes. When/why is it needed? We could either drop this or add a check before it for whether the stage has changed.

[dberenbaum]

Where is the data being imported from?

It doesn't matter. I have tried with local and cloud remotes paths and get the same behavior.

[dberenbaum]

Raising to p0 since this impacts anyone doing updates across all remote types.