external outputs: broken if pipeline output doesn't exist during stage initialization

[dberenbaum]

Bug Report

Description

S3 external outputs are broken for pipelines since 7211bd0 because of a bug in s3fs (and probably in other filesystems). They will only break if running a stage for which an output doesn't already exist. When initializing the stage, DVC will try to remove the nonexistent output and raise a FileNotFound error.

Reproduce

dvc repro will break if there is an external output and that output does not exist yet.

In a new repo, using some <s3_path> that doesn't exist yet, do this:

$ echo 'foo' > foo
$ dvc stage add --external -n foo -d foo -O <s3_path> 'aws s3 cp params.yaml <s3_path>'
$ dvc repro -v

Expected

dvc repro shouldn't fail while removing outputs. In this case, it fails because of what seems like a bug or at least inconsistent behavior in fsspec. Like mentioned in #5961 (comment), output.remove for s3fs and other async filesystems calls _expand_path. When the path doesn't exist and recursive=True, _expand_path raises FileNotFoundError. When recursive=False, it returns the path. It also returns the path for the LocalFileSystem regardless of whether recursive=True, so not sure if it was intended to raise an error only for this specific scenario.

[dberenbaum]

Not sure if this should be a new issue, but I'm seeing the same problem now because we have recursive=True in

dvc/dvc/output.py

Line 968 in 8f768ad

self.fs.remove(self.fs_path, recursive=True)

.

[dberenbaum]

@daavoo Do you recall from #8825 if we need recursive=True?

[daavoo]

@daavoo Do you recall from #8825 if we need recursive=True?

I didn't test its removal but it was supposed to be a legacy option no longer needed iterative/dvc-objects#176 (comment)

[daavoo]

So @dberenbaum to see if I got it right, is this because fsspec raises FileNotFound error?
I don't know if this is a bug in fsspec or expected behavior

[dberenbaum]

Correct. It also wasn't clear to me whether it's intended.