New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Azure Gateway to azure-storage-blob SDK #8537
Conversation
Can you resolve the conflicts @c-w ? thanks |
@harshavardhana: rebased onto master and fixed the merge conflicts. |
PTAL @vadmeste @Praveenrajmani |
The azure-sdk-for-go/storage package has been in maintenance-only mode since February 2018 (see [1]) and will be deprecated in the future. Additionally, there are several limitations with the storage SDK, including lack of support for containerized storage deployments like Azurite [2] or Azure IoT Edge storage [3]. These limitations currently make it hard to work with the MinIO Azure Gateway in disconnected scenarios, e.g. for local development or local integration testing. As such, this change updates the MinIO Azure Gateway to use the new and recommended Azure Storage SDK: azure-storage-blob-go/azblob [4]. Besides migrating away from a package that is no longer actively developed and adding support for containerized storage deployment backends, this new SDK also enables us to in many cases combine previously separate PutBlob, SetMetadata and SetProperties requests into a single call which should slightly improve performance. [1] Azure/azure-sdk-for-go@fa0a4d1 [2] https://github.com/Azure/Azurite [3] https://docs.microsoft.com/en-us/azure/iot-edge/how-to-store-data-blob [4] https://github.com/Azure/azure-storage-blob-go
To validate all these changes please also run |
@harshavardhana Thanks for letting me know that I should run mint with the edge tag. I previously only ran mint:latest and those tests passed (see screenshot I linked from the pull request description). I noticed that when I run mint:edge against the latest published minio docker image I get a failure in the aws-cli tests: docker rmi minio/minio:latest
docker pull minio/minio:latest
MINIO_ACCESS_KEY=changeme
az group create -l eastus -n $MINIO_ACCESS_KEY
az storage account create -l eastus --sku Standard_LRS --kind StorageV2 -n $MINIO_ACCESS_KEY -g $MINIO_ACCESS_KEY
MINIO_SECRET_KEY=$(az storage account keys list -n $MINIO_ACCESS_KEY -o tsv --query [0].value)
docker network create minio
docker run -d --rm --network minio --name minio -e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY -e MINIO_SECRET_KEY=$MINIO_SECRET_KEY minio/minio:latest gateway azure
docker run --network minio -e SERVER_ENDPOINT=minio:9000 -e ACCESS_KEY=$MINIO_ACCESS_KEY -e SECRET_KEY=$MINIO_SECRET_KEY minio/mint:edge The docker logs show the following: Can you reproduce this failure on your end or is there something that I'm doing wrong? |
Also, make sure to set MINT_MODE=full to run more / bigger tests @c-w |
This looks like PutObjectPart code needs to be fixed, its missing content-length header @c-w |
@harshavardhana Thanks for the reply. In my previous comment I mentioned that the error I posted is happening for the latest published MinIO image, not my branch. It looks like this is an existing issue with MinIO. To double check and confirm, I executed the tests on a completely clean VM pulling only I suspect the cause of the issue is this line in master which allows for zero-length blocks to be uploaded (this gets rejected by the Azure API). I have fixed this issue in my branch in 5a6f830, however, I'd hypothesize that this existing error reduces the usefulness of the mint tests in evaluating this pull request since we don't seem to have a stable test suite to compare against. Please advise how you'd like to proceed. |
Thanks @c-w, I will take a look with mint fails with the standard MinIO image. Also I noticed something weird while testing your PR. If you pass wrong credentials, some operations like, creating bucket, putting object won't fail with 403 as it used to be.
Then run this in another terminal (after configuring mc alias)
|
@c-w I am not able to reproduce the mint error, can you tell us how did you start your MinIO server container ? |
@vadmeste I used the bash snippet in #8537 (comment) to run MinIO and Mint. |
@vadmeste Great catch. Turns out the case where we get an Azure error from the SDK but don't convert it to a MinIO error wasn't covered by the unit tests so the refactor introduced a regression. I fixed the behavior and also added a test to cover the scenario in 32e94e2. |
Mint Automation
8537-7289ccb/mint-compression-xl.sh.log:
8537-7289ccb/mint-large-bucket.sh.log:
8537-7289ccb/mint-compression-dist-xl.sh.log:
Error: No such image: minio/minio:8537-7289ccb |
PTAL @vadmeste |
Has this issue been addressed? I have the same problem |
The azure-sdk-for-go/storage package has been in maintenance- only mode since February 2018 (see [1]) and will be deprecated in the future.
Description
This change updates the MinIO Azure Gateway to use the new Azure Storage SDK: azure-storage-blob-go/azblob.
Motivation and Context
The currently used Azure Storage SDK azure-sdk-for-go/storage has been in maintenance-only mode since February 2018 and will be deprecated in the future.
Additionally, there are several limitations with the SDK, including lack of support for containerized storage deployments like Azurite or Azure IoT Edge storage. These limitations currently make it hard to work with the MinIO Azure Gateway in disconnected scenarios, e.g. for local development or local integration testing.
Besides migrating away from an SDK that is no longer actively developed and adding support for containerized storage deployment backends, this change also enables us to in many cases combine previously separate PutBlob, SetMetadata and SetProperties requests into a single call which should slightly improve performance.
How to test this PR?
First, build the code from this change into a Docker image, e.g. via
TAG=minio/minio make docker
. Then, execute the mint tests against the updated Azure gateway using Azurite as the backend storage implementation and observe that all the tests pass:Types of changes
Checklist: