Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AzCopyV10][Bug] Uploading from top directory on linux fails when sub folder has the same name #1762

Closed
wants to merge 7 commits into from

Conversation

siminsavani-msft
Copy link
Member

@siminsavani-msft siminsavani-msft commented May 5, 2022

Line 19 in cmd/copyEnumeratorHelper.go causes a bug in linux in which the root folder name is removed from path.

i.e. e.SourceRoot.Value = "/event" and transfer.Source = "/event_stuff/foo.txt" --> incorrectly changes transfer.Source to "_stuff/foo.txt"

The transfer.Source is already a relative path and this line of code can be removed.


Refactoring AzCopy copy to use scheduleCopyTransfer instead of addTransfer.

Steps of scheduleCopyTransfer:

  1. Gets source and destination relative path from stored object (transfer)
  2. Sets up copy transfer and figures out if stored object should be sent to STE
  3. If destination is none, then blobtier, metadata, and blob tags are set accordingly
  4. If the stored object should not be sent to STE, it skips this object
  5. If transfer has dry run mode on, it logs info and skips the object
  6. If the length of transfers is same as number of transfers per part, then reset transfers buffer
  7. Append transfer to list after checking and dispatching a part
  8. If transfer type is file, increment file count or else increment folder count

Steps of addTransfer:

  1. Trims prefix for source and destination url (to get relative paths for both)
  2. Dispatch transfers once number of files reaches the number of files per dispatch job part (10000) and reset transfers buffer.
  3. Append transfer to transfer list after a part has been checked and dispatched
  4. If transfer type is file, increment file count or else increment folder count

@zezha-msft zezha-msft added this to the 10.16.0 milestone Jun 30, 2022
@zezha-msft zezha-msft modified the milestones: 10.16.0, 10.18.0 Oct 10, 2022
// Remove the destination roots from the path to save space in the plan files

// TODO: Remove this code because transfer.Source will already have relative path
// transfer.Source = strings.TrimPrefix(transfer.Source, e.SourceRoot.Value)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is transfer.Destination not a similar case? Does this bug end up existing on the destination?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call out! I removed the transfer.Destination (in the new draft PR) as well because transfer.Destination will always have relative path.

@siminsavani-msft
Copy link
Member Author

Closing this PR and creating fresh PR here: #2125

@siminsavani-msft siminsavani-msft removed this from the 10.18.0 milestone Mar 23, 2023
@siminsavani-msft siminsavani-msft deleted the siminsavani/azcopy-linux-bug branch March 23, 2023 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants