GH-38705: [C++][FS][Azure] Implement CopyFile()

[kou]

Rationale for this change

CopyFile() copies the given source to the given destination. Both of source and destination must be blob name like other filesystem implementations.

What changes are included in this PR?

Use CopyFromUri() API that should use server-side copy.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

• Closes: [C++][FS][Azure] Implement CopyFile() #38705

[github-actions]

⚠️ GitHub issue #38705 has been automatically assigned in GitHub to PR creator.

[kou]

@Tom-Newton @felipecrv What do you think about this behavior? I think that this is convenient but this is different from other filesystem implementations...

If the given destination has only container (container) or has trailing slash (container/directory/), base name of the given source is automatically appended. For example:

• Source: container1/directory1/path.txt
• Destination: container2
• Real destination: container2/path.txt (path.txt is only appended to container2)

This is the same behavior as cp command. But other filesystem implementations require path not directory as destination.

[Tom-Newton]

This behaviour sounds fine to me.

I think there is a problem with the CopyFromUri() API though. I believe it requires using SAS token auth on the source, because the source URI needs to be self authenticated.

[Tom-Newton]

I think there is a problem with the CopyFromUri() API though.

Nevermind, it looks like you've got a working implementation using CopyFromUri(). Maybe it does auth automatically if it detects that the source is in the same storage account. Or possibly the Azure SDK automatically switches between Copy Blob From URL and Copy Blob

[pitrou]

I think that this is convenient but this is different from other filesystem implementations...

We should not diverge from the filesystem spec (as described in the FileSystem class docstrings ).
The end goal is for all implementations to expose a similar behavior.

Also, at some point the Azure fs implementation will have to implement and pass the generic filesystem tests, which do have a test for this situation.

[felipecrv]

I think that this is convenient but this is different from other filesystem implementations...

We should not diverge from the filesystem spec (as described in the FileSystem class docstrings ). The end goal is for all implementations to expose a similar behavior.

Also, at some point the Azure fs implementation will have to implement and pass the generic filesystem tests, which do have a test for this situation.

I agree. I think we should fix #38772 before adding more operations. I implemented the GetFileInfo(select) with this as an assumption. CopyFile shouldn't implicitly create directories, so we need a way to check an empty directory exists.

[kou]

Oh, sorry. I didn't read the docstring. I'll remove the behavior.

Also, at some point the Azure fs implementation will have to implement and pass the generic filesystem tests, which do have a test for this situation.

I've opened a new issue for it: #39069

[kou]

I think there is a problem with the CopyFromUri() API though.

Nevermind, it looks like you've got a working implementation using CopyFromUri(). Maybe it does auth automatically if it detects that the source is in the same storage account. Or possibly the Azure SDK automatically switches between Copy Blob From URL and Copy Blob

I think that "Copy Blob From URL" and "Copy Blob" are the same API internally.
Both of them use the same URI and PUT:

• Copy Blob From URL: https://learn.microsoft.com/en-us/rest/api/storageservices/copy-blob-from-url#request
• Copy Blob: https://learn.microsoft.com/en-us/rest/api/storageservices/copy-blob#request

And BlobClient provides only CopyFromUri(). (It doesn't provide Copy().)
So I used CopyFromUri().

[kou]

Updated:

• Accept only blob path for destination like other implementations.

[kou]

I'll merge this in a few days if nobody objects it.

[Tom-Newton]

LGTM

[Tom-Newton]

Personally I'm not particularly worried but to be rigorous @felipecrv's comment should probably be addressed.

CopyFile shouldn't implicitly create directories, so we need a way to check an empty directory exists.

[kou]

OK. I've implemented it.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 70ccf33.

There were 10 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2023-12-07 14:23:01Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-06, scale_factor=1
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-06, scale_factor=10
• and 8 more (see the report linked below)

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.