Skip to content

Commit

Permalink
Add env var for rsakeypassphrase
Browse files Browse the repository at this point in the history
- Migrate changelog
  • Loading branch information
alfpark committed Jul 28, 2016
1 parent e5e435a commit 11b79a5
Show file tree
Hide file tree
Showing 4 changed files with 161 additions and 63 deletions.
150 changes: 150 additions & 0 deletions CHANGELOG.md
@@ -0,0 +1,150 @@
# Change Log

## [Unreleased]

## [0.11.2] - 2016-07-28
### Added
- Allow rsakeypassphrase to be passed as an environment variable

## 0.11.1 - 2016-07-05
### Added
- Allow storage account or sas key credentials to be passed as
environment variables

## 0.11.0 - 2016-06-09
### Added
- Azure Files support, please refer to the General Notes section for
limitations

### Changed
- `--blobep` option has been renamed to `--endpoint`

## 0.10.1 - 2016-06-06
### Changed
- Update all dependencies to latest versions
- Add flag for block/page level md5 computation which is now disabled by
default

### Fixed
- Update against breaking changes from azure-storage 0.32.0

### Removed
- Remove RC designation from encryption/decryption functionality

### Security
- Update cryptography requirement to 1.4

## 0.10.0 - 2016-03-22
### Added
- Added ``--disable-urllib-warnings`` option to suppress urllib3 warnings
(use with care)

### Changed
- Update script for compatibility with azure-storage 0.30.0 which
is now a required dependency
- Promote encryption to RC status
- `--blobep` now refers to endpoint suffix rather than blob endpoint
(e.g., core.windows.net rather than blob.core.windows.net)

### Security
- Update cryptography requirement to 1.3

## 0.9.9.11 - 2016-02-22
### Fixed
- Minor bug fixes

## Changed
- Pin azure dependencies due to breaking changes

### Security
- Update cryptography requirement to 1.2.2

## 0.9.9.10 - 2016-01-31
### Fixed
- Fix regression in blob name encoding with Python3

## 0.9.9.9 - 2016-01-29
### Added
- Emit warning when attempting to use remoteresource with a directory upload

### Changed
- Update setup.py dependencies to latest available versions

### Fixed
- Fix regression in single file upload and remoteresource renaming
- Replace socket exception handling with requests ConnectionError handling
- Properly handle blob names containing `?` if using SAS

## 0.9.9.8 - 2016-01-06
### Fixed
- Disable unnecessary thread daemonization
- Gracefully handle KeyboardInterrupts
- Explicitly add azure-common to setup.py install reqs

## 0.9.9.7 - 2016-01-05
### Added
- Add python environment and package info to parameter dump to aid issue/bug
reports

### Changed
- Reduce number of default concurrent workers to 3x CPU count
- Change azure\_request backoff mechanism

### Fixed
- Make base requirements non-optional in import process
- Update azure\_request exception handling to support new Azure Storage Python
SDK errors

## 0.9.9.6 - 2016-01-04
### Added
- Encryption support
- No file overwrite on download option
- Auto-detection of file mimetype
- Remote delete option
- Include pattern option

### Changed
- Replace keeprootdir with strip-components option
- Reduce the number of default concurrent workers to 4x CPU count

### Fixed
- Fix shared key upload with non-existent container
- Fix zero-byte blob download issue

## 0.9.9.5 - 2015-09-27
### Added
- File collation support

### Fixed
- Fix page alignment bug
- Reduce memory usage

## Old Releases
- 0.9.9.4: improve page blob upload algorithm to skip empty max size pages.
fix zero length file uploads. fix single file upload that's skipped.
- 0.9.9.3: fix downloading of blobs with content length of zero
- 0.9.9.1: fix content length > 32bit for blob lists via SAS on Python2
- 0.9.9.0: update script for compatibility with new Azure Python packages
- 0.9.8: fix blob endpoint for non-SAS input, add retry on ServerBusy
- 0.9.7: normalize SAS keys (accept keys with or without ? char prefix)
- 0.9.6: revert local resource path expansion, PEP8 fixes
- 0.9.5: fix directory creation issue
- 0.9.4: fix Python3 compatibility issues
- 0.9.3: the script supports page blob uploading. To specify local files to
upload as page blobs, specify the `--pageblob` parameter. The script also
has a feature to detect files ending in the `.vhd` extension and will
automatically upload just these files as page blobs while uploading other
files as block blobs. Specify the `--autovhd` parameter (without the
`--pageblob` parameter) to enable this behavior.
- 0.9.0: the script will automatically default to skipping files where if the
MD5 checksum of either the local file or the stored MD5 of the remote
resource respectively matches the remote resource or local file, then the
upload or download for the file will be skipped. This capability will allow
one to perform rsync-like operations where only files that have changed will
be transferred. This behavior can be forcefully disabled by specifying
`--no-skiponmatch`.
- 0.8.2: performance regression fixes

[Unreleased]: https://github.com/Azure/blobxfer/compare/v0.11.2...HEAD
[0.11.2]: https://github.com/Azure/blobxfer/compare/e5e435a...v0.11.2

63 changes: 3 additions & 60 deletions README.rst
Expand Up @@ -368,63 +368,6 @@ Encryption Notes
Change Log
----------

- 0.11.1: Allow storage account or sas key credentials to be passed as
environment variables.
- 0.11.0: Azure Files support, please refer to the General Notes section for
limitations. ``--blobep`` option has been renamed to ``--endpoint``.
- 0.10.1: remove RC designation from encryption/decryption functionality.
update all dependencies to latest versions. update against breaking changes
from azure-storage 0.32.0. add flag for block/page level md5 computation
which is now disabled by default.
- 0.10.0: update script for compatibility with azure-storage 0.30.0 which
is now a required dependency, update cryptography requirement to 1.3,
promote encryption to RC status, ``--blobep`` now refers to endpoint suffix
rather than blob endpoint (e.g., core.windows.net rather than
blob.core.windows.net), added ``--disable-urllib-warnings`` option to
suppress urllib3 warnings (use with care)
- 0.9.9.11: minor bug fixes, update cryptography requirement to 1.2.2, pin
azure dependencies due to breaking changes
- 0.9.9.10: fix regression in blob name encoding with Python3
- 0.9.9.9: fix regression in single file upload and remoteresource renaming,
emit warning when attempting to use remoteresource with a directory upload,
replace socket exception handling with requests ConnectionError handling,
properly handle blob names containing ``?`` if using SAS, update setup.py
dependencies to latest available versions
- 0.9.9.8: disable unnecessary thread daemonization, gracefully handle
KeyboardInterrupts, explicitly add azure-common to setup.py install reqs
- 0.9.9.7: make base requirements non-optional in import process, update
azure_request exception handling to support new Azure Storage Python SDK
errors, reduce number of default concurrent workers to 3x CPU count, change
azure_request backoff mechanism, add python environment and package info to
parameter dump to aid issue/bug reports
- 0.9.9.6: add encryption support, fix shared key upload with non-existent
container, add file overwrite on download option, add auto-detection of file
mimetype, add remote delete option, fix zero-byte blob download issue,
replace keeprootdir with strip-components option, add include option,
reduce the number of default concurrent workers to 4x CPU count
- 0.9.9.5: add file collation support, fix page alignment bug, reduce memory
usage
- 0.9.9.4: improve page blob upload algorithm to skip empty max size pages.
fix zero length file uploads. fix single file upload that's skipped.
- 0.9.9.3: fix downloading of blobs with content length of zero
- 0.9.9.1: fix content length > 32bit for blob lists via SAS on Python2
- 0.9.9.0: update script for compatibility with new Azure Python packages
- 0.9.8: fix blob endpoint for non-SAS input, add retry on ServerBusy
- 0.9.7: normalize SAS keys (accept keys with or without ? char prefix)
- 0.9.6: revert local resource path expansion, PEP8 fixes
- 0.9.5: fix directory creation issue
- 0.9.4: fix Python3 compatibility issues
- 0.9.3: the script supports page blob uploading. To specify local files to
upload as page blobs, specify the ``--pageblob`` parameter. The script also
has a feature to detect files ending in the ``.vhd`` extension and will
automatically upload just these files as page blobs while uploading other
files as block blobs. Specify the ``--autovhd`` parameter (without the
``--pageblob`` parameter) to enable this behavior.
- 0.9.0: the script will automatically default to skipping files where if the
MD5 checksum of either the local file or the stored MD5 of the remote
resource respectively matches the remote resource or local file, then the
upload or download for the file will be skipped. This capability will allow
one to perform rsync-like operations where only files that have changed will
be transferred. This behavior can be forcefully disabled by specifying
``--no-skiponmatch``.
- 0.8.2: performance regression fixes
See the `CHANGELOG.md`_ file.

.. _CHANGELOG.md: https://github.com/Azure/blobxfer/CHANGELOG.md
9 changes: 7 additions & 2 deletions blobxfer.py
Expand Up @@ -104,7 +104,7 @@
# pylint: enable=W0622,C0103

# global defines
_SCRIPT_VERSION = '0.11.1'
_SCRIPT_VERSION = '0.11.2'
_PY2 = sys.version_info.major == 2
_DEFAULT_MAX_STORAGEACCOUNT_WORKERS = multiprocessing.cpu_count() * 3
_MAX_BLOB_CHUNK_SIZE_BYTES = 4194304
Expand All @@ -115,6 +115,7 @@
_DEFAULT_MANAGEMENT_ENDPOINT = 'management.core.windows.net'
_ENVVAR_STORAGEACCOUNTKEY = 'BLOBXFER_STORAGEACCOUNTKEY'
_ENVVAR_SASKEY = 'BLOBXFER_SASKEY'
_ENVVAR_RSAKEYPASSPHRASE = 'BLOBXFER_RSAKEYPASSPHRASE'
# encryption defines
_AES256_KEYLENGTH_BYTES = 32
_AES256_BLOCKSIZE_BYTES = 16
Expand Down Expand Up @@ -2014,6 +2015,8 @@ def main():
args.storageaccountkey = os.getenv(_ENVVAR_STORAGEACCOUNTKEY)
if args.saskey is None:
args.saskey = os.getenv(_ENVVAR_SASKEY)
if args.rsakeypassphrase is None:
args.rsakeypassphrase = os.getenv(_ENVVAR_RSAKEYPASSPHRASE)

# check some parameters
if (len(args.localresource) < 1 or len(args.storageaccount) < 1 or
Expand Down Expand Up @@ -2952,7 +2955,9 @@ def parseargs(): # pragma: no cover
'for encrypting and uploading blobs.')
parser.add_argument(
'--rsakeypassphrase',
help='Optional passphrase for decrypting an RSA private key.')
help='Optional passphrase for decrypting an RSA private key; can be '
'specified as {} environment variable instead'.format(
_ENVVAR_RSAKEYPASSPHRASE))
parser.add_argument(
'--remoteresource',
help='name of remote resource on Azure storage. "."=container '
Expand Down
2 changes: 1 addition & 1 deletion setup.py
Expand Up @@ -20,7 +20,7 @@
description='Azure storage transfer tool with AzCopy-like features',
long_description=long_description,
platforms='any',
url='https://github.com/Azure/azure-batch-samples/Python/Storage',
url='https://github.com/Azure/blobxfer',
license='MIT',
py_modules=['blobxfer'],
entry_points={
Expand Down

0 comments on commit 11b79a5

Please sign in to comment.