Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Add mitigation for weird NtQuerySecurityObject behavior on NAS sources (#1872) * Add check for 0 length, attempt to validate the returned object. * Change to grabbing real SD length * Add comment describing issue * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fix bad URL delete (#1892) * Manipulate URLs safely * Fix folder deletion test * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Update MacOS testing pipeline (#1896) * fixing small typo (,) in help of jobs clean (#1899) * Microsoft mandatory file * fixing small typo (,) in help of jobs clean Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com> * Implement MD OAuth testing (#1859) * Implement MD OAuth testing * Handle async on RevokeAccess, handle job cancel/failure better * Prevent parallel testing of managed disks * lint check * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Add env vars * Avoid revoking MD access, as it can be shared. * Fix intermittent failures * Disable MD OAuth testing temporarily. * Add "all" to documentation (#1902) * 10.16.1 patch notes (#1913) * Add bugfixes to change log. * Correct wording & punctuation * Correct version * Export Successfully Updated bytes (#1884) * Add info in error message for mkdir on Log/Plan (#1883) * Microsoft mandatory file * Add info in error message for mkdir on Log/Plan Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com> * Fix fixupTokenJson (#1890) * Microsoft mandatory file * Fix fixupTokenJson Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com> Co-authored-by: Adam Orosz <adam.orosz@neotechnology.com> * Do not log request/response for container creation error (#1893) * Expose AZCOPY_DOWNLOAD_TO_TEMP_PATH environment variable. (#1895) * Slice against the correct string (#1927) * UX improvement: avoid crash when copying S2S with user delegation SAS (#1932) * Fix bad build + Prevent bad builds in the future (#1917) * Fix bad build + Prevent bad builds in the future * Add Windows build * Make sync use last write time for Azure Files (#1930) * Make sync use last write time for Azure Files * Implement test * 10.16.2 Changelog (#1948) * Update azcopy version * Fixed a bug where preserve permissions would not work with OAuth * Added CODEOWNERS file * Fixed issue where CPK would not be injected on retries * remove OAuth from test * Updated version check string to indicate current AzCopy version (#1969) * added codeowner * Enhance job summary with details about file/folders (#1952) * Add flag to disable version check (#1950) * darwin arm64 * Update golang version to 10.19.2 (#1925) * enable cgo * added tests * Minor fixes: More in description (#1968) * Echo auto-login failure if any * Update help for sync command to use trailing slash on directories * azcopy fail to copy 12TB file to Storage containers in Dev. The logic is used to calculate proper blockSize if it’s not provided, and due to the uint32 cast, it can’t give proper blockSize if filesize is between 50000 * (8 * 1024 * 1024) * X + 1, to 50000 * (8 * 1024 * 1024) * X + 49999. It should return 16MB instead of 8MB blockSize. Accommodated the changes suggested by Narasimha Kulkarni * Added extra logging when switching endpoints * Enable support for preserving SMB info on Linux. (#1723) * Microsoft mandatory file * Enable support for preserving SMB info on Linux. Implemented the GetSDDL/PutSDDL GetSMBProperties/PutSMBProperties methods for Linux using extended attributes. Following are the xattrs we use for fetching/setting various required info. // Extended Attribute (xattr) keys for fetching various information from Linux cifs client. const ( CIFS_XATTR_CREATETIME = "user.cifs.creationtime" // File creation time. CIFS_XATTR_ATTRIB = "user.cifs.dosattrib" // FileAttributes. CIFS_XATTR_CIFS_ACL = "system.cifs_acl" // DACL only. CIFS_XATTR_CIFS_NTSD = "system.cifs_ntsd" // Owner, Group, DACL. CIFS_XATTR_CIFS_NTSD_FULL = "system.cifs_ntsd_full" // Owner, Group, DACL, SACL. ) Majority of the changes are in sddl/sddlHelper_linux.go which implement the following Win32 APIs for dealing with SIDs. ConvertSecurityDescriptorToStringSecurityDescriptorW ConvertStringSecurityDescriptorToSecurityDescriptorW ConvertSidToStringSidW ConvertStringSidToSidW Note: I have skipped Object ACE support in sddl/sddlHelper_linux.go as those should not be used for filesystem properties, only AD object properties. Can someone confirm this? TBD: Conditional SID * Audited, fixed, tested support for "No ACL"/NO_ACCESS_CONTROL and ACL w/o any ACE Tested the following cases: c:\Users\natomar\Downloads>cd testacl // This has "No ACLs" and everyone should be allowed access. c:\Users\natomar\Downloads\testacl>touch NO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls NO_ACCESS_CONTROL.txt /S:D:NO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\NO_ACCESS_CONTROL.txt // This has "No ACLs" and everyone should be allowed access. // It additionally has the "P" (protected) flag set, but that won't have // any effect as that just prevents ACE inheritance but this ACL will // not have any ACLs due to the NO_ACCESS_CONTROL flag. c:\Users\natomar\Downloads\testacl>touch PNO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls PNO_ACCESS_CONTROL.txt /S:D:PNO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\PNO_ACCESS_CONTROL.txt // This should set DACL but with no ACEs, but since "P" is not set it // inherits ACEs from the parent dir. c:\Users\natomar\Downloads\testacl>touch empty_d.txt c:\Users\natomar\Downloads\testacl>cacls empty_d.txt /S:D: Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d.txt // This should set DACL but with no ACEs, but since "P" is set it // doesn't inherit ACEs from the parent dir and hence this will block // all users. c:\Users\natomar\Downloads\testacl>touch empty_d_with_p.txt c:\Users\natomar\Downloads\testacl>cacls empty_d_with_p.txt /S:D:P Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d_with_p.txt * Don't fail outright for ACL revision 4. Though our supported ACL types must carry ACL revision 2 as per the doc https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/20233ed8-a6c6-4097-aafa-dd545ed24428 but I've seen some dirs have ACL revision 4 but ACL types are still supported ones. So instead of failing upfront, let it fail with unsupported ACE type. Also hexadecimal aceRights are more commonly seen than I expected, so removing a log. * Minor fix after running azcopy on a large dir. This was something which I have doubt on. Now that we got a real world issue due to this, it's all clear :-) * Some minor updates after the rebase to latest Azcopy. * Set default value of flag preserve-smb-info to true on Windows and false on other OS (cherry picked from commit ac5bedb) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Nagendra Tomar <Nagendra.Tomar@microsoft.com> * Added log indicating a sub-directory is being enqueued (#1999) * Log sync deletions to scanning logger (#2000) * ieproxy fix * remove cgo * fix * fix * fix * more testing * more testing * more testing * more testing * mod tidy * mod tidy * more testing * Added codespell (#2008) * Added codespell * Fixed initial codespell errors * Fix format in codespell.yml * Added s3 url parts * Added CodeQL (#2009) * Added linting file * Upgrade codeql to v2 * Fix incorrect conversion between integer types * Fix GCP URL parts * Fix for rare infinite loop on mutex acquisition (#2012) * small fix * removed test * Added trivy file (#2015) * Added trivy file * renamed trivy * Improve debug-ability of e2e tests by uploading logs of failed jobs (#1898) * Upload testing logs to storage account on failed test * Handle as pipeline artifact instead * mkdirall * copy plan files too * Fix failing tests * Change overwrite to affect any "locked in"/completed state * Fail copy job if single blob does not exist (#1981) * Job fail if single file does not exist * fixed change * fail only on a single file not existing * fail on file not found * fail on file not found * fail on file not found * cleanup * added tests * cleanup * removed test * Correct odd behavior around folder overwrites (#1961) * Fix files sync by determining which LMT to use via smb properties flag (#1958) * Fix files sync by determining which LMT to use via smb properties flag * Implement testing for LMT switch * Fix testing * Limit SMB testing to SMB-compatible environment * Enforce SMB LMT for Linux/MacOS test of SMB LMT preference * Fix metadata parsing (#1953) * Fix metadata parsing * rework metadata parsing to be more robust; add test * Fix comment lines * Codespell :| * Fix ADLSG2 intermittent failure (#1901) * Fix ADLSG2 intermittent failure * Add test * Reduce code dupe * Fix build errors * Fix infinite loop maybe? * Store source token and pass to other threads (#1996) * Store source token * testing * failing pipe * cleanup * test logger * fix test failure * fix 2 * fix * sync fix * cleanup check * Hash based sync (#2020) * Implement hash based sync for MD5 * Implement testing * Ensure folders are handled properly in HBS & Test S2S * Add skip/process logging * Include generic xattr syncmeta application * Fix 0-size blobs * Fix core testing * Revert "Include generic xattr syncmeta application" This reverts commit fba55e4. * Warn on no hash @ source, remove MHP * Comments * Comments * Copy properties from Source (#1964) * Copy properties from Source * Remove unnecessary ws changes * Preserve UNIX properties * Move entity type to Overwrite option * Add python suite * Review comments * Fix test * Release notes and version update (#2028) Co-authored-by: adreed-msft <49764384+adreed-msft@users.noreply.github.com> Co-authored-by: mstenz <mstenz-design@web.de> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com> Co-authored-by: Adele Reed <adreed@microsoft.com> Co-authored-by: Karla Saur <1703543+ksaur@users.noreply.github.com> Co-authored-by: adam-orosz <106535811+adam-orosz@users.noreply.github.com> Co-authored-by: Adam Orosz <adam.orosz@neotechnology.com> Co-authored-by: Ze Qian Zhang <zezha@microsoft.com> Co-authored-by: Gauri Prasad <gapra@microsoft.com> Co-authored-by: Gauri Prasad <51212198+gapra-msft@users.noreply.github.com> Co-authored-by: Tamer Sherif <tasherif@microsoft.com> Co-authored-by: Tamer Sherif <69483382+tasherif-msft@users.noreply.github.com> Co-authored-by: reshmav18 <73923840+reshmav18@users.noreply.github.com> Co-authored-by: linuxsmiths <linuxsmiths@gmail.com> Co-authored-by: Nagendra Tomar <Nagendra.Tomar@microsoft.com>
- Loading branch information
17 people
committed
Jan 23, 2023
1 parent
a10fdd0
commit 108dbdd