Skip to content

Commit

Permalink
10.18.0 release (#2136)
Browse files Browse the repository at this point in the history
* Add mitigation for weird NtQuerySecurityObject behavior on NAS sources (#1872)

* Add check for 0 length, attempt to validate the returned object.

* Change to grabbing real SD length

* Add comment describing issue

* Prevent infinite loop upon listing failure

* Fix GCP error checking

* Fix GCP disable

* Fix bad URL delete (#1892)

* Manipulate URLs safely

* Fix folder deletion test

* Prevent infinite loop upon listing failure

* Fix GCP error checking

* Fix GCP disable

* Fail when errors listing/clearing bucket

* Update MacOS testing pipeline (#1896)

* fixing small typo (,) in help of jobs clean (#1899)

* Microsoft mandatory file

* fixing small typo (,) in help of jobs clean

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>

* Implement MD OAuth testing (#1859)

* Implement MD OAuth testing

* Handle async on RevokeAccess, handle job cancel/failure better

* Prevent parallel testing of managed disks

* lint check

* Prevent infinite loop upon listing failure

* Fix GCP error checking

* Fix GCP disable

* Fail when errors listing/clearing bucket

* Add env vars

* Avoid revoking MD access, as it can be shared.

* Fix intermittent failures

* Disable MD OAuth testing temporarily.

* Add "all" to documentation (#1902)

* 10.16.1 patch notes (#1913)

* Add bugfixes to change log.

* Correct wording & punctuation

* Correct version

* Export Successfully Updated bytes (#1884)

* Add info in error message for mkdir on Log/Plan (#1883)

* Microsoft mandatory file

* Add info in error message for mkdir on Log/Plan

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>

* Fix fixupTokenJson (#1890)

* Microsoft mandatory file

* Fix fixupTokenJson

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>
Co-authored-by: Adam Orosz <adam.orosz@neotechnology.com>

* Do not log request/response for container creation error (#1893)

* Expose AZCOPY_DOWNLOAD_TO_TEMP_PATH environment variable. (#1895)

* Slice against the correct string (#1927)

* UX improvement: avoid crash when copying S2S with user delegation SAS (#1932)

* Fix bad build + Prevent bad builds in the future (#1917)

* Fix bad build + Prevent bad builds in the future

* Add Windows build

* Make sync use last write time for Azure Files (#1930)

* Make sync use last write time for Azure Files

* Implement test

* 10.16.2 Changelog (#1948)

* Update azcopy version

* Fixed a bug where preserve permissions would not work with OAuth

* Added CODEOWNERS file

* Fixed issue where CPK would not be injected on retries

* remove OAuth from test

* Updated version check string to indicate current AzCopy version (#1969)

* added codeowner

* Enhance job summary with details about file/folders (#1952)

* Add flag to disable version check (#1950)

* darwin arm64

* Update golang version to 10.19.2 (#1925)

* enable cgo

* added tests

* Minor fixes: More in description (#1968)

* Echo auto-login failure if any

* Update help for sync command to use trailing slash on directories

* azcopy fail to copy 12TB file to Storage containers in Dev.

The logic is used to calculate proper blockSize if it’s not provided, and due to the uint32 cast, it can’t give proper blockSize if filesize is between 50000 * (8 * 1024 * 1024) * X + 1, to 50000 * (8 * 1024 * 1024) * X + 49999. It should return 16MB instead of 8MB blockSize.

Accommodated the changes suggested by Narasimha Kulkarni

* Added extra logging when switching endpoints

* Enable support for preserving SMB info on Linux. (#1723)

* Microsoft mandatory file

* Enable support for preserving SMB info on Linux.

Implemented the GetSDDL/PutSDDL GetSMBProperties/PutSMBProperties
methods for Linux using extended attributes.
Following are the xattrs we use for fetching/setting various required
info.

// Extended Attribute (xattr) keys for fetching various information from Linux cifs client.
const (
        CIFS_XATTR_CREATETIME     = "user.cifs.creationtime" // File creation time.
        CIFS_XATTR_ATTRIB         = "user.cifs.dosattrib"    // FileAttributes.
        CIFS_XATTR_CIFS_ACL       = "system.cifs_acl"        // DACL only.
        CIFS_XATTR_CIFS_NTSD      = "system.cifs_ntsd"       // Owner, Group, DACL.
        CIFS_XATTR_CIFS_NTSD_FULL = "system.cifs_ntsd_full"  // Owner, Group, DACL, SACL.
)

Majority of the changes are in sddl/sddlHelper_linux.go which implement
the following Win32 APIs for dealing with SIDs.

	ConvertSecurityDescriptorToStringSecurityDescriptorW
	ConvertStringSecurityDescriptorToSecurityDescriptorW
	ConvertSidToStringSidW
	ConvertStringSidToSidW

Note: I have skipped Object ACE support in sddl/sddlHelper_linux.go as
      those should not be used for filesystem properties, only AD object
      properties.
      Can someone confirm this?

TBD:
Conditional SID

* Audited, fixed, tested support for "No ACL"/NO_ACCESS_CONTROL and ACL w/o any ACE

Tested the following cases:

c:\Users\natomar\Downloads>cd testacl

// This has "No ACLs" and everyone should be allowed access.
c:\Users\natomar\Downloads\testacl>touch NO_ACCESS_CONTROL.txt
c:\Users\natomar\Downloads\testacl>cacls NO_ACCESS_CONTROL.txt /S:D:NO_ACCESS_CONTROL
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\NO_ACCESS_CONTROL.txt

// This has "No ACLs" and everyone should be allowed access.
// It additionally has the "P" (protected) flag set, but that won't have
// any effect as that just prevents ACE inheritance but this ACL will
// not have any ACLs due to the NO_ACCESS_CONTROL flag.
c:\Users\natomar\Downloads\testacl>touch PNO_ACCESS_CONTROL.txt
c:\Users\natomar\Downloads\testacl>cacls PNO_ACCESS_CONTROL.txt /S:D:PNO_ACCESS_CONTROL
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\PNO_ACCESS_CONTROL.txt

// This should set DACL but with no ACEs, but since "P" is not set it
// inherits ACEs from the parent dir.
c:\Users\natomar\Downloads\testacl>touch empty_d.txt
c:\Users\natomar\Downloads\testacl>cacls empty_d.txt /S:D:
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\empty_d.txt

// This should set DACL but with no ACEs, but since "P" is set it
//  doesn't inherit ACEs from the parent dir and hence this will block
// all users.
c:\Users\natomar\Downloads\testacl>touch empty_d_with_p.txt
c:\Users\natomar\Downloads\testacl>cacls empty_d_with_p.txt /S:D:P
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\empty_d_with_p.txt

* Don't fail outright for ACL revision 4.

Though our supported ACL types must carry ACL revision 2 as per the doc

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/20233ed8-a6c6-4097-aafa-dd545ed24428

but I've seen some dirs have ACL revision 4 but ACL types are still
supported ones. So instead of failing upfront, let it fail with
unsupported ACE type.

Also hexadecimal aceRights are more commonly seen than I expected, so
removing a log.

* Minor fix after running azcopy on a large dir.

This was something which I have doubt on. Now that we got a real world
issue due to this, it's all clear :-)

* Some minor updates after the rebase to latest Azcopy.

* Set default value of flag preserve-smb-info to true on Windows and false on other OS

(cherry picked from commit ac5bedb)

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Nagendra Tomar <Nagendra.Tomar@microsoft.com>

* Added log indicating a sub-directory is being enqueued (#1999)

* Log sync deletions to scanning logger (#2000)

* ieproxy fix

* remove cgo

* fix

* fix

* fix

* more testing

* more testing

* more testing

* more testing

* mod tidy

* mod tidy

* more testing

* Added codespell (#2008)

* Added codespell

* Fixed initial codespell errors

* Fix format in codespell.yml

* Added s3 url parts

* Added CodeQL (#2009)

* Added linting file

* Upgrade codeql to v2

* Fix incorrect conversion between integer types

* Fix GCP URL parts

* Fix for rare infinite loop on mutex acquisition (#2012)

* small fix

* removed test

* Added trivy file (#2015)

* Added trivy file

* renamed trivy

* Improve debug-ability of e2e tests by uploading logs of failed jobs (#1898)

* Upload testing logs to storage account on failed test

* Handle as pipeline artifact instead

* mkdirall

* copy plan files too

* Fix failing tests

* Change overwrite to affect any "locked in"/completed state

* Fail copy job if single blob does not exist (#1981)

* Job fail if single file does not exist

* fixed change

* fail only on a single file not existing

* fail on file not found

* fail on file not found

* fail on file not found

* cleanup

* added tests

* cleanup

* removed test

* Correct odd behavior around folder overwrites (#1961)

* Fix files sync by determining which LMT to use via smb properties flag (#1958)

* Fix files sync by determining which LMT to use via smb properties flag

* Implement testing for LMT switch

* Fix testing

* Limit SMB testing to SMB-compatible environment

* Enforce SMB LMT for Linux/MacOS test of SMB LMT preference

* Fix metadata parsing (#1953)

* Fix metadata parsing

* rework metadata parsing to be more robust; add test

* Fix comment lines

* Codespell :|

* Fix ADLSG2 intermittent failure (#1901)

* Fix ADLSG2 intermittent failure

* Add test

* Reduce code dupe

* Fix build errors

* Fix infinite loop maybe?

* Store source token and pass to other threads (#1996)

* Store source token

* testing

* failing pipe

* cleanup

* test logger

* fix test failure

* fix 2

* fix

* sync fix

* cleanup check

* Hash based sync (#2020)

* Implement hash based sync for MD5

* Implement testing

* Ensure folders are handled properly in HBS & Test S2S

* Add skip/process logging

* Include generic xattr syncmeta application

* Fix 0-size blobs

* Fix core testing

* Revert "Include generic xattr syncmeta application"

This reverts commit fba55e4.

* Warn on no hash @ source, remove MHP

* Comments

* Comments

* Copy properties from Source (#1964)

* Copy properties from Source

* Remove unnecessary ws changes

* Preserve UNIX properties

* Move entity type to Overwrite option

* Add python suite

* Review comments

* Fix test

* Release notes and version update (#2028)

* Fix SIGSERV on MacOS causes by mattn/go-ieproxy#35. Fixes #1790. (#1993)

* Revert "Fix SIGSERV on MacOS causes by mattn/go-ieproxy#35. Fixes #1790. (#1993)" (#2044)

This reverts commit dc7666a.

* Fixed HTML Blob Type (#2048)

* fixed typos

* typo

* Implement upload/download of symlinks (#1829)

* Implement symlink upload/download

* Fix tests

* Handle comments

* Testing

* Fix test arguments

* Fix files MD5 issues

* Limit symlink persistence to relevant locations

* Update block ID generation logic (#2050)

* Change BlockID generation Logic

* Add testcase

* Add comment to explain naming of block IDS

* Reformt blockID name convention

* Upgrade GCS storage library (#2064)

* Update GCS module

* Increase lint timeout

* Set GC to default (#2063)

* Fix incorrectly displayed environment variable name for plan file path

* Updating helpMessages.go to further clarify the --compare-hash flag

* Add golangci-lint (#2010)

* Upgrade linting version (#2105)

* Implement container-level ACL copies (#2049)

* Implement container-level ACL copies

* Prevent AssertNoErr from crashing test suite

* Fix delete functionality & testing

* Grab subdir in a different way

* Fix Build

* Lint

* Reimplement POSIX properties download (#1835)

* Reimplement POSIX properties download

* Fix upload/download of special file types (pipe, fifo, dev, cdev)

* Posix Download testing

+ Fix for symlink download

* Prevent folders from generating while using CPK

* Remove removed bit

* Fix tests, add ADLSG2 tests

* Limit scope of POSIX test metadata

* Golint

* Fix wildcard handling when a file of the same name as the wildcard input exists (#2062)

* Fix stgexp bug

* Fix test compilation

* Upgrade azcopy dependencies (#2115)

* Option to transfer properties-only in jobPart (#2071)

* Added a scenario to the help messages (#2121)

* Allow specifying "Cold" tier (#2096)

* Allow specifying "Cold" tier

* Switch service version when setting tier

* Do not create parent directory if it is root (#2120)

* Do not create parent directory if it is root

* Add testcase

* Add testcase2

* Resume transfer of incomplete file (#2119)

* Resume file

* iIntroduce transfer status restarted

* Spelling mistake

* Add resume to uploads

* Add environment variable and make log msg more clear

* Change log level to Debug for a few statements (#2123)

* Change log level to Debug for a few statements

* Fix repeated messages on empty input

* Perf testing for AzCopy (#2006)

* Base commit for perf test

* Add tests

* Add local tests

* Re-enable S2S and use /dev/null for local tests

* Add auth mode to Clean destination Containers

* Modify bench command to have runtime around 30min

* Change image name

* Add auth mode to cli command

* Change large files count to 50

* Redact CPK headers in log output. (#2127)

Currently, azcopy will emit the following headers to log files, when using Customer Provided Keys:

x-ms-encryption-key
x-ms-encryption-key-sha256

I believe this is unexpected for some users. This patch redacts those headers from log files, such
that encryption keys do not accidentally leak.

Co-authored-by: Mikkel Krautz <mk@intertisement.com>

* [AzCopyV10]  Add in content-md5 to available list properties (#2033)

* Release 10.17.0 (#2029)

* Add mitigation for weird NtQuerySecurityObject behavior on NAS sources (#1872)

* Add check for 0 length, attempt to validate the returned object.

* Change to grabbing real SD length

* Add comment describing issue

* Prevent infinite loop upon listing failure

* Fix GCP error checking

* Fix GCP disable

* Fix bad URL delete (#1892)

* Manipulate URLs safely

* Fix folder deletion test

* Prevent infinite loop upon listing failure

* Fix GCP error checking

* Fix GCP disable

* Fail when errors listing/clearing bucket

* Update MacOS testing pipeline (#1896)

* fixing small typo (,) in help of jobs clean (#1899)

* Microsoft mandatory file

* fixing small typo (,) in help of jobs clean

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>

* Implement MD OAuth testing (#1859)

* Implement MD OAuth testing

* Handle async on RevokeAccess, handle job cancel/failure better

* Prevent parallel testing of managed disks

* lint check

* Prevent infinite loop upon listing failure

* Fix GCP error checking

* Fix GCP disable

* Fail when errors listing/clearing bucket

* Add env vars

* Avoid revoking MD access, as it can be shared.

* Fix intermittent failures

* Disable MD OAuth testing temporarily.

* Add "all" to documentation (#1902)

* 10.16.1 patch notes (#1913)

* Add bugfixes to change log.

* Correct wording & punctuation

* Correct version

* Export Successfully Updated bytes (#1884)

* Add info in error message for mkdir on Log/Plan (#1883)

* Microsoft mandatory file

* Add info in error message for mkdir on Log/Plan

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>

* Fix fixupTokenJson (#1890)

* Microsoft mandatory file

* Fix fixupTokenJson

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>
Co-authored-by: Adam Orosz <adam.orosz@neotechnology.com>

* Do not log request/response for container creation error (#1893)

* Expose AZCOPY_DOWNLOAD_TO_TEMP_PATH environment variable. (#1895)

* Slice against the correct string (#1927)

* UX improvement: avoid crash when copying S2S with user delegation SAS (#1932)

* Fix bad build + Prevent bad builds in the future (#1917)

* Fix bad build + Prevent bad builds in the future

* Add Windows build

* Make sync use last write time for Azure Files (#1930)

* Make sync use last write time for Azure Files

* Implement test

* 10.16.2 Changelog (#1948)

* Update azcopy version

* Fixed a bug where preserve permissions would not work with OAuth

* Added CODEOWNERS file

* Fixed issue where CPK would not be injected on retries

* remove OAuth from test

* Updated version check string to indicate current AzCopy version (#1969)

* added codeowner

* Enhance job summary with details about file/folders (#1952)

* Add flag to disable version check (#1950)

* darwin arm64

* Update golang version to 10.19.2 (#1925)

* enable cgo

* added tests

* Minor fixes: More in description (#1968)

* Echo auto-login failure if any

* Update help for sync command to use trailing slash on directories

* azcopy fail to copy 12TB file to Storage containers in Dev.

The logic is used to calculate proper blockSize if it’s not provided, and due to the uint32 cast, it can’t give proper blockSize if filesize is between 50000 * (8 * 1024 * 1024) * X + 1, to 50000 * (8 * 1024 * 1024) * X + 49999. It should return 16MB instead of 8MB blockSize.

Accommodated the changes suggested by Narasimha Kulkarni

* Added extra logging when switching endpoints

* Enable support for preserving SMB info on Linux. (#1723)

* Microsoft mandatory file

* Enable support for preserving SMB info on Linux.

Implemented the GetSDDL/PutSDDL GetSMBProperties/PutSMBProperties
methods for Linux using extended attributes.
Following are the xattrs we use for fetching/setting various required
info.

// Extended Attribute (xattr) keys for fetching various information from Linux cifs client.
const (
        CIFS_XATTR_CREATETIME     = "user.cifs.creationtime" // File creation time.
        CIFS_XATTR_ATTRIB         = "user.cifs.dosattrib"    // FileAttributes.
        CIFS_XATTR_CIFS_ACL       = "system.cifs_acl"        // DACL only.
        CIFS_XATTR_CIFS_NTSD      = "system.cifs_ntsd"       // Owner, Group, DACL.
        CIFS_XATTR_CIFS_NTSD_FULL = "system.cifs_ntsd_full"  // Owner, Group, DACL, SACL.
)

Majority of the changes are in sddl/sddlHelper_linux.go which implement
the following Win32 APIs for dealing with SIDs.

	ConvertSecurityDescriptorToStringSecurityDescriptorW
	ConvertStringSecurityDescriptorToSecurityDescriptorW
	ConvertSidToStringSidW
	ConvertStringSidToSidW

Note: I have skipped Object ACE support in sddl/sddlHelper_linux.go as
      those should not be used for filesystem properties, only AD object
      properties.
      Can someone confirm this?

TBD:
Conditional SID

* Audited, fixed, tested support for "No ACL"/NO_ACCESS_CONTROL and ACL w/o any ACE

Tested the following cases:

c:\Users\natomar\Downloads>cd testacl

// This has "No ACLs" and everyone should be allowed access.
c:\Users\natomar\Downloads\testacl>touch NO_ACCESS_CONTROL.txt
c:\Users\natomar\Downloads\testacl>cacls NO_ACCESS_CONTROL.txt /S:D:NO_ACCESS_CONTROL
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\NO_ACCESS_CONTROL.txt

// This has "No ACLs" and everyone should be allowed access.
// It additionally has the "P" (protected) flag set, but that won't have
// any effect as that just prevents ACE inheritance but this ACL will
// not have any ACLs due to the NO_ACCESS_CONTROL flag.
c:\Users\natomar\Downloads\testacl>touch PNO_ACCESS_CONTROL.txt
c:\Users\natomar\Downloads\testacl>cacls PNO_ACCESS_CONTROL.txt /S:D:PNO_ACCESS_CONTROL
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\PNO_ACCESS_CONTROL.txt

// This should set DACL but with no ACEs, but since "P" is not set it
// inherits ACEs from the parent dir.
c:\Users\natomar\Downloads\testacl>touch empty_d.txt
c:\Users\natomar\Downloads\testacl>cacls empty_d.txt /S:D:
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\empty_d.txt

// This should set DACL but with no ACEs, but since "P" is set it
//  doesn't inherit ACEs from the parent dir and hence this will block
// all users.
c:\Users\natomar\Downloads\testacl>touch empty_d_with_p.txt
c:\Users\natomar\Downloads\testacl>cacls empty_d_with_p.txt /S:D:P
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\empty_d_with_p.txt

* Don't fail outright for ACL revision 4.

Though our supported ACL types must carry ACL revision 2 as per the doc

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/20233ed8-a6c6-4097-aafa-dd545ed24428

but I've seen some dirs have ACL revision 4 but ACL types are still
supported ones. So instead of failing upfront, let it fail with
unsupported ACE type.

Also hexadecimal aceRights are more commonly seen than I expected, so
removing a log.

* Minor fix after running azcopy on a large dir.

This was something which I have doubt on. Now that we got a real world
issue due to this, it's all clear :-)

* Some minor updates after the rebase to latest Azcopy.

* Set default value of flag preserve-smb-info to true on Windows and false on other OS

(cherry picked from commit ac5bedb)

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Nagendra Tomar <Nagendra.Tomar@microsoft.com>

* Added log indicating a sub-directory is being enqueued (#1999)

* Log sync deletions to scanning logger (#2000)

* ieproxy fix

* remove cgo

* fix

* fix

* fix

* more testing

* more testing

* more testing

* more testing

* mod tidy

* mod tidy

* more testing

* Added codespell (#2008)

* Added codespell

* Fixed initial codespell errors

* Fix format in codespell.yml

* Added s3 url parts

* Added CodeQL (#2009)

* Added linting file

* Upgrade codeql to v2

* Fix incorrect conversion between integer types

* Fix GCP URL parts

* Fix for rare infinite loop on mutex acquisition (#2012)

* small fix

* removed test

* Added trivy file (#2015)

* Added trivy file

* renamed trivy

* Improve debug-ability of e2e tests by uploading logs of failed jobs (#1898)

* Upload testing logs to storage account on failed test

* Handle as pipeline artifact instead

* mkdirall

* copy plan files too

* Fix failing tests

* Change overwrite to affect any "locked in"/completed state

* Fail copy job if single blob does not exist (#1981)

* Job fail if single file does not exist

* fixed change

* fail only on a single file not existing

* fail on file not found

* fail on file not found

* fail on file not found

* cleanup

* added tests

* cleanup

* removed test

* Correct odd behavior around folder overwrites (#1961)

* Fix files sync by determining which LMT to use via smb properties flag (#1958)

* Fix files sync by determining which LMT to use via smb properties flag

* Implement testing for LMT switch

* Fix testing

* Limit SMB testing to SMB-compatible environment

* Enforce SMB LMT for Linux/MacOS test of SMB LMT preference

* Fix metadata parsing (#1953)

* Fix metadata parsing

* rework metadata parsing to be more robust; add test

* Fix comment lines

* Codespell :|

* Fix ADLSG2 intermittent failure (#1901)

* Fix ADLSG2 intermittent failure

* Add test

* Reduce code dupe

* Fix build errors

* Fix infinite loop maybe?

* Store source token and pass to other threads (#1996)

* Store source token

* testing

* failing pipe

* cleanup

* test logger

* fix test failure

* fix 2

* fix

* sync fix

* cleanup check

* Hash based sync (#2020)

* Implement hash based sync for MD5

* Implement testing

* Ensure folders are handled properly in HBS & Test S2S

* Add skip/process logging

* Include generic xattr syncmeta application

* Fix 0-size blobs

* Fix core testing

* Revert "Include generic xattr syncmeta application"

This reverts commit fba55e4.

* Warn on no hash @ source, remove MHP

* Comments

* Comments

* Copy properties from Source (#1964)

* Copy properties from Source

* Remove unnecessary ws changes

* Preserve UNIX properties

* Move entity type to Overwrite option

* Add python suite

* Review comments

* Fix test

* Release notes and version update (#2028)

Co-authored-by: adreed-msft <49764384+adreed-msft@users.noreply.github.com>
Co-authored-by: mstenz <mstenz-design@web.de>
Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>
Co-authored-by: Adele Reed <adreed@microsoft.com>
Co-authored-by: Karla Saur <1703543+ksaur@users.noreply.github.com>
Co-authored-by: adam-orosz <106535811+adam-orosz@users.noreply.github.com>
Co-authored-by: Adam Orosz <adam.orosz@neotechnology.com>
Co-authored-by: Ze Qian Zhang <zezha@microsoft.com>
Co-authored-by: Gauri Prasad <gapra@microsoft.com>
Co-authored-by: Gauri Prasad <51212198+gapra-msft@users.noreply.github.com>
Co-authored-by: Tamer Sherif <tasherif@microsoft.com>
Co-authored-by: Tamer Sherif <69483382+tasherif-msft@users.noreply.github.com>
Co-authored-by: reshmav18 <73923840+reshmav18@users.noreply.github.com>
Co-authored-by: linuxsmiths <linuxsmiths@gmail.com>
Co-authored-by: Nagendra Tomar <Nagendra.Tomar@microsoft.com>

* Add in content md5

* Change variable name

* Change to base64 encoding

* Update list.go

* Fix the object for md5.

* Fix name to mathc help and formating.

* Switch to decode to string

* Fix syntax error.

* Left of b

* Undoing conversion

* Fix syntax error.

* Fixed camelcase.

* Fix syntax error.

* Fix syntax error.

---------

Co-authored-by: Narasimha Kulkarni <nakulkar@microsoft.com>
Co-authored-by: adreed-msft <49764384+adreed-msft@users.noreply.github.com>
Co-authored-by: mstenz <mstenz-design@web.de>
Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>
Co-authored-by: Adele Reed <adreed@microsoft.com>
Co-authored-by: Karla Saur <1703543+ksaur@users.noreply.github.com>
Co-authored-by: adam-orosz <106535811+adam-orosz@users.noreply.github.com>
Co-authored-by: Adam Orosz <adam.orosz@neotechnology.com>
Co-authored-by: Ze Qian Zhang <zezha@microsoft.com>
Co-authored-by: Gauri Prasad <gapra@microsoft.com>
Co-authored-by: Gauri Prasad <51212198+gapra-msft@users.noreply.github.com>
Co-authored-by: Tamer Sherif <tasherif@microsoft.com>
Co-authored-by: Tamer Sherif <69483382+tasherif-msft@users.noreply.github.com>
Co-authored-by: reshmav18 <73923840+reshmav18@users.noreply.github.com>
Co-authored-by: linuxsmiths <linuxsmiths@gmail.com>
Co-authored-by: Nagendra Tomar <Nagendra.Tomar@microsoft.com>

* Fail early if we detect invalid URL on one of targets (#2128)

* Fail early if we detect invalid URL on one of targets

* Add testcase

* [AzCopyV10][Bug] Uploading from top directory on linux fails when sub folder has the same name (#2125)

* Adding TODOs and bug fix seen with linux subfolders

* Add test for relative path and commented out old test

* Removed TODOs and cleaned up comments

* 10.18 changelog (#2135)

* Make golang version configurable (#2137)

* Make golang version configurable

* Make golang version configurable

* Update azure-pipelines.yml for Azure Pipelines

* Update azure-pipelines.yml for Azure Pipelines

* Increment version #

---------

Co-authored-by: mstenz <mstenz-design@web.de>
Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>
Co-authored-by: Narasimha Kulkarni <nakulkar@microsoft.com>
Co-authored-by: Karla Saur <1703543+ksaur@users.noreply.github.com>
Co-authored-by: adam-orosz <106535811+adam-orosz@users.noreply.github.com>
Co-authored-by: Adam Orosz <adam.orosz@neotechnology.com>
Co-authored-by: Ze Qian Zhang <zezha@microsoft.com>
Co-authored-by: Gauri Prasad <gapra@microsoft.com>
Co-authored-by: Gauri Prasad <51212198+gapra-msft@users.noreply.github.com>
Co-authored-by: Tamer Sherif <tasherif@microsoft.com>
Co-authored-by: Tamer Sherif <69483382+tasherif-msft@users.noreply.github.com>
Co-authored-by: reshmav18 <73923840+reshmav18@users.noreply.github.com>
Co-authored-by: linuxsmiths <linuxsmiths@gmail.com>
Co-authored-by: Nagendra Tomar <Nagendra.Tomar@microsoft.com>
Co-authored-by: Richard Kettelerij <291572+rkettelerij@users.noreply.github.com>
Co-authored-by: Norm Estabrook <normesta@microsoft.com>
Co-authored-by: Mikkel Krautz <mk@intertisement.com>
Co-authored-by: Venkat Malladi <vsmalladi@gmail.com>
Co-authored-by: siminsavani-msft <77068571+siminsavani-msft@users.noreply.github.com>
  • Loading branch information
21 people committed Mar 31, 2023
1 parent 108dbdd commit 053a049
Show file tree
Hide file tree
Showing 162 changed files with 3,294 additions and 1,243 deletions.
46 changes: 23 additions & 23 deletions .github/workflows/codespell.yml
@@ -1,24 +1,24 @@
# GitHub Action to automate the identification of common misspellings in text files.
# https://github.com/codespell-project/actions-codespell
# https://github.com/codespell-project/codespell
name: codespell
on:
push:
branches:
- dev
- main
pull_request:
branches:
- dev
- main
jobs:
codespell:
name: Check for spelling errors
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: codespell-project/actions-codespell@master
with:
check_filenames: true
skip: ./sddl/sddlPortable_test.go,./sddl/sddlHelper_linux.go
# GitHub Action to automate the identification of common misspellings in text files.
# https://github.com/codespell-project/actions-codespell
# https://github.com/codespell-project/codespell
name: codespell
on:
push:
branches:
- dev
- main
pull_request:
branches:
- dev
- main
jobs:
codespell:
name: Check for spelling errors
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: codespell-project/actions-codespell@master
with:
check_filenames: true
skip: ./sddl/sddlPortable_test.go,./sddl/sddlHelper_linux.go,./go.sum
ignore_words_list: "resue,pase,cancl,cacl,froms"
40 changes: 40 additions & 0 deletions .github/workflows/golangci-lint.yml
@@ -0,0 +1,40 @@
name: golangci-lint
on:
push:
branches:
- dev
- main
pull_request:
branches:
- dev
- main
permissions:
contents: read
# Optional: allow read access to pull request. Use with `only-new-issues` option.
# pull-requests: read
jobs:
golangci:
strategy:
matrix:
go: [1.18]
os: [ubuntu-latest, windows-latest, macos-latest]
name: lint
runs-on: ${{ matrix.os }}
steps:
- uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go }}
- uses: actions/checkout@v3
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: latest
# Optional: working directory, useful for monorepos
# working-directory: somedir

# Optional: golangci-lint command line arguments.
args: --tests=false --max-issues-per-linter=0 --skip-files=azbfs/zz_generated_* --skip-dirs=e2etest --exclude=S1008 --max-same-issues=0 --timeout 5m0s

# Optional: show only new issues if it's a pull request. The default value is `false`.
# only-new-issues: true
26 changes: 26 additions & 0 deletions ChangeLog.md
@@ -1,6 +1,32 @@

# Change Log

## Version 10.18.0

### New features

1. Added support for `Content-MD5` in `list` command. User can now list the MD5 hash of the blobs in the target container.
2. Added support to resume incomplete blobs. User can now resume the upload of a blob which was interrupted in the middle.
3. Added support for download of POSIX properties.
4. Added support for persisting symlink data.

### Bug fixes

1. Fixed [Issue 2120](https://github.com/Azure/azure-storage-azcopy/pull/2120)
2. Fixed [Issue 2062](https://github.com/Azure/azure-storage-azcopy/pull/2062)
3. Fixed [Issue 2046](https://github.com/Azure/azure-storage-azcopy/pull/2048)
4. Fixed [Issue 1762](https://github.com/Azure/azure-storage-azcopy/pull/2125)

### Documentation

1. Added example for `--include-pattern`.
2. Added documentation for `--compare-hash`.

### Security fixes

1. CPK-related headers are now sanitized from the logs.
2. Updated dependencies to address security vulnerabilities.

## Version 10.17.0

### New features
Expand Down
13 changes: 4 additions & 9 deletions azbfs/zc_credential_anonymous.go
Expand Up @@ -12,15 +12,6 @@ type Credential interface {
credentialMarker()
}

type credentialFunc pipeline.FactoryFunc

func (f credentialFunc) New(next pipeline.Policy, po *pipeline.PolicyOptions) pipeline.Policy {
return f(next, po)
}

// credentialMarker is a package-internal method that exists just to satisfy the Credential interface.
func (credentialFunc) credentialMarker() {}

//////////////////////////////

// NewAnonymousCredential creates an anonymous credential for use with HTTP(S) requests that read public resource
Expand All @@ -36,11 +27,15 @@ type anonymousCredentialPolicyFactory struct {
}

// New creates a credential policy object.
//nolint:unused
func (f *anonymousCredentialPolicyFactory) New(next pipeline.Policy, po *pipeline.PolicyOptions) pipeline.Policy {
// Note: We are not deleting this "unused" code since this is a publicly exported function, we do not want to break
// anyone that has a dependency on the azbfs library (like blobfuse).
return &anonymousCredentialPolicy{next: next}
}

// credentialMarker is a package-internal method that exists just to satisfy the Credential interface.
//nolint:unused
func (*anonymousCredentialPolicyFactory) credentialMarker() {}

// anonymousCredentialPolicy is the credential's policy object.
Expand Down
34 changes: 14 additions & 20 deletions azbfs/zc_credential_shared_key.go
Expand Up @@ -62,25 +62,19 @@ func (f *SharedKeyCredential) New(next pipeline.Policy, po *pipeline.PolicyOptio
// credentialMarker is a package-internal method that exists just to satisfy the Credential interface.
func (*SharedKeyCredential) credentialMarker() {}

// Constants ensuring that header names are correctly spelled and consistently cased.
const (
headerAuthorization = "Authorization"
headerCacheControl = "Cache-Control"
headerContentEncoding = "Content-Encoding"
headerContentDisposition = "Content-Disposition"
headerContentLanguage = "Content-Language"
headerContentLength = "Content-Length"
headerContentMD5 = "Content-MD5"
headerContentType = "Content-Type"
headerDate = "Date"
headerIfMatch = "If-Match"
headerIfModifiedSince = "If-Modified-Since"
headerIfNoneMatch = "If-None-Match"
headerIfUnmodifiedSince = "If-Unmodified-Since"
headerRange = "Range"
headerUserAgent = "User-Agent"
headerXmsDate = "x-ms-date"
headerXmsVersion = "x-ms-version"
headerAuthorization = "Authorization"
headerContentEncoding = "Content-Encoding"
headerContentLanguage = "Content-Language"
headerContentLength = "Content-Length"
headerContentMD5 = "Content-MD5"
headerContentType = "Content-Type"
headerIfMatch = "If-Match"
headerIfModifiedSince = "If-Modified-Since"
headerIfNoneMatch = "If-None-Match"
headerIfUnmodifiedSince = "If-Unmodified-Since"
headerRange = "Range"
headerXmsDate = "x-ms-date"
)

// ComputeHMACSHA256 generates a hash signature for an HTTP request or for a SAS.
Expand Down Expand Up @@ -143,7 +137,7 @@ func buildCanonicalizedHeader(headers http.Header) string {
ch.WriteRune(':')
ch.WriteString(strings.Join(cm[key], ","))
}
return string(ch.Bytes())
return ch.String()
}

func (f *SharedKeyCredential) buildCanonicalizedResource(u *url.URL) string {
Expand Down Expand Up @@ -190,5 +184,5 @@ func (f *SharedKeyCredential) buildCanonicalizedResource(u *url.URL) string {
cr.WriteString("\n" + paramName + ":" + strings.Join(paramValues, ","))
}
}
return string(cr.Bytes())
return cr.String()
}
27 changes: 0 additions & 27 deletions azbfs/zc_mmf_unix.go

This file was deleted.

39 changes: 0 additions & 39 deletions azbfs/zc_mmf_windows.go

This file was deleted.

4 changes: 2 additions & 2 deletions azbfs/zc_policy_request_log.go
Expand Up @@ -76,9 +76,9 @@ func NewRequestLogPolicyFactory_Deprecated(o RequestLogOptions) pipeline.Factory
sc := response.Response().StatusCode
if ((sc >= 400 && sc <= 499) && sc != http.StatusNotFound && sc != http.StatusConflict && sc != http.StatusPreconditionFailed && sc != http.StatusRequestedRangeNotSatisfiable) || (sc >= 500 && sc <= 599) {
logLevel, forceLog = pipeline.LogError, !o.SyslogDisabled // Promote to Error any 4xx (except those listed is an error) or any 5xx
} else {
// For other status codes, we leave the level as is.
}
// For other status codes, we leave the level as is.

} else { // This error did not get an HTTP response from the service; upgrade the severity to Error
logLevel, forceLog = pipeline.LogError, !o.SyslogDisabled
}
Expand Down
9 changes: 4 additions & 5 deletions azbfs/zc_policy_retry.go
Expand Up @@ -3,7 +3,6 @@ package azbfs
import (
"context"
"io"
"io/ioutil"
"math/rand"
"net"
"net/http"
Expand Down Expand Up @@ -134,7 +133,7 @@ func (o RetryOptions) calcDelay(try int32) time.Duration { // try is >=1; never
// Note: forked from the standard package url.go
// The content is exactly the same but the spaces are encoded as %20 instead of +
// TODO: remove after the service fix
// Encode encodes the values into ``URL encoded'' form
// Encode encodes the values into URL encoded form
// ("bar=baz&foo=quux") sorted by key.
func alternativeEncode(v url.Values) string {
if v == nil {
Expand Down Expand Up @@ -217,7 +216,7 @@ func NewRetryPolicyFactory(o RetryOptions) pipeline.Factory {
// Set the server-side timeout query parameter "timeout=[seconds]"
timeout := int32(o.TryTimeout.Seconds()) // Max seconds per try
if deadline, ok := ctx.Deadline(); ok { // If user's ctx has a deadline, make the timeout the smaller of the two
t := int32(deadline.Sub(time.Now()).Seconds()) // Duration from now until user's ctx reaches its deadline
t := int32(time.Until(deadline).Seconds()) // Duration from now until user's ctx reaches its deadline
logf("MaxTryTimeout=%d secs, TimeTilDeadline=%d sec\n", timeout, t)
if t < timeout {
timeout = t
Expand Down Expand Up @@ -254,7 +253,7 @@ func NewRetryPolicyFactory(o RetryOptions) pipeline.Factory {
action = "Retry: Secondary URL returned 404"
case err != nil:
// NOTE: Protocol Responder returns non-nil if REST API returns invalid status code for the invoked operation
if netErr, ok := err.(net.Error); ok && (netErr.Temporary() || netErr.Timeout()) {
if netErr, ok := err.(net.Error); ok && (netErr.Temporary() || netErr.Timeout()) { //nolint:staticcheck
action = "Retry: net.Error and Temporary() or Timeout()"
} else if err == io.ErrUnexpectedEOF {
// Some of our methods under the zz_ files do use io.Copy and other related methods that can throw an unexpectedEOF.
Expand Down Expand Up @@ -284,7 +283,7 @@ func NewRetryPolicyFactory(o RetryOptions) pipeline.Factory {
}
if response != nil && response.Response() != nil {
// If we're going to retry and we got a previous response, then flush its body to avoid leaking its TCP connection
io.Copy(ioutil.Discard, response.Response().Body)
_, _ = io.Copy(io.Discard, response.Response().Body)
response.Response().Body.Close()
}
// If retrying, cancel the current per-try timeout context
Expand Down
4 changes: 2 additions & 2 deletions azbfs/zc_sas_query_params.go
Expand Up @@ -200,7 +200,7 @@ func newSASQueryParameters(values url.Values, deleteSASParametersFromValues bool
case "st":
p.startTime, p.stTimeFormat, _ = parseSASTimeString(val)
case "se":
p.expiryTime, p.stTimeFormat, _ = parseSASTimeString(val)
p.expiryTime, p.seTimeFormat, _ = parseSASTimeString(val)
case "sip":
dashIndex := strings.Index(val, "-")
if dashIndex == -1 {
Expand Down Expand Up @@ -255,7 +255,7 @@ func (p *SASQueryParameters) addToValues(v url.Values) url.Values {
v.Add("st", formatSASTime(&p.startTime, p.stTimeFormat))
}
if !p.expiryTime.IsZero() {
v.Add("se", formatSASTime(&p.expiryTime, p.stTimeFormat))
v.Add("se", formatSASTime(&p.expiryTime, p.seTimeFormat))
}
if len(p.ipRange.Start) > 0 {
v.Add("sip", p.ipRange.String())
Expand Down
5 changes: 4 additions & 1 deletion azbfs/zc_util_validate.go
Expand Up @@ -58,7 +58,10 @@ func validateSeekableStreamAt0AndGetCount(body io.ReadSeeker) int64 {
if err != nil {
panic("failed to seek stream")
}
body.Seek(0, io.SeekStart)
_, err = body.Seek(0, io.SeekStart)
if err != nil {
logf("error seeking stream (%s)", err.Error())
}
return count
}

Expand Down

0 comments on commit 053a049

Please sign in to comment.