Validation optimization - reducing the number of HEAD calls during cp and mv operations

[zveinn]

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the [Apache 2 license] (https://www.apache.org/licenses/LICENSE-2.0).
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

This pull request provides performance optimizations for the mv and cp commands along with some minor cleanup and validation refactor.

Related Issues

#3149

Motivation and Context

During some cp and mv operations the mc client makes too many HEAD calls. These head calls were being made on multiple levels of execution so in order to reduce the overhead a sizable cleanup was required.

The Validation layer

The main offender here was the checkCopySyntax function. On Line 61 the function validates object existance, however, the objects existance is validate numerous times through out the process making this redundant. The next offender start on line 111, this is the initial syntax validation. This validation method calls for HEAD multiple times to validate the syntax.

Merging URL generation and Syntax Validation

The simplest way to reduce the number of HEAD calls is to no repeat that operation multiple times for the same object. This means passing already known information down the execution path. The simplest way to achieve this goal was to merge the validation and URL generation layer since the URL generation layer has it's own requirement on HEAD checking. It would not have been wise to merge the URL generation layer up into the validation layer since the logic behind the URL generation is much more complex.

This is an example trace of a file being moved from a local filesystem to a bucket

2023-10-10T08:20:35.958 [200 OK] s3.GetBucketLocation 192.168.2.10:9000/bb1/?location=  192.168.2.10     176µs       ⇣  165.972µs  ↑ 77 B ↓ 128 B
2023-10-10T08:20:35.959 [200 OK] s3.HeadObject 192.168.2.10:9000/bb1/m 192.168.2.10     6.663ms      ⇣  0s         ↑ 77 B ↓ 0 B
2023-10-10T08:20:35.967 [200 OK] s3.HeadObject 192.168.2.10:9000/bb1/m 192.168.2.10     200µs       ⇣  0s         ↑ 77 B ↓ 0 B
2023-10-10T08:20:35.967 [200 OK] s3.HeadObject 192.168.2.10:9000/bb1/m 192.168.2.10     209µs       ⇣  0s         ↑ 77 B ↓ 0 B
2023-10-10T08:20:35.968 [200 OK] s3.HeadObject 192.168.2.10:9000/bb1/m 192.168.2.10     190µs       ⇣  0s         ↑ 77 B ↓ 0 B
2023-10-10T08:20:35.969 [200 OK] s3.HeadObject 192.168.2.10:9000/bb1/m 192.168.2.10     174µs       ⇣  0s         ↑ 77 B ↓ 0 B
2023-10-10T08:20:35.970 [200 OK] s3.HeadObject 192.168.2.10:9000/bb1/m 192.168.2.10     174µs       ⇣  0s         ↑ 77 B ↓ 0 B
2023-10-10T08:20:35.971 [200 OK] s3.GetObject 192.168.2.10:9000/bb1/m 192.168.2.10     24.561ms     ⇣  1.507702ms  ↑ 86 B ↓ 25 MiB

Example of the same operation post refactor

2023-10-10T08:40:18.971 [200 OK] s3.GetBucketLocation 192.168.2.10:9000/bb1/?location=  192.168.2.10     241µs       ⇣  224.181µs  ↑ 77 B ↓ 128 B
2023-10-10T08:40:18.971 [200 OK] s3.HeadObject 192.168.2.10:9000/bb1/m 192.168.2.10     306µs       ⇣  0s         ↑ 77 B ↓ 0 B
2023-10-10T08:40:18.972 [200 OK] s3.HeadObject 192.168.2.10:9000/bb1/m 192.168.2.10     280µs       ⇣  0s         ↑ 77 B ↓ 0 B
2023-10-10T08:40:18.973 [200 OK] s3.GetObject 192.168.2.10:9000/bb1/m 192.168.2.10     24.231ms     ⇣  1.320942ms  ↑ 86 B ↓ 25 MiB

How to test this PR?

There is a functional-tests.sh script in the root of the project.

NOTE: The test_admin_users functional test seems to be broken and I could not figure out why it's not working. It has nothing to do with my current changes and I suspect it's been broken for a while. I did refactor the test_admin_users test a bit and I opened a discussion on slack about it.

$ ./functional-tests.sh

Types of changes

• Optimization (provides speedup with no functional changes)

Checklist:

• Unit tests added/updated

[klauspost]

Just some minor things.

[klauspost]

LGTM