Generating blob SAS tokens is failing with the NuGet package of version 12.20.0 and Azurite 3.30.0 #2406

Arash-Sabet · 2024-06-03T13:26:50Z

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

3.30.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

npm

What's the Node.js version?

Presumably the latest LTS version

What problem was encountered?

Steps to reproduce the issue?

We are including azurite in our docker image by the following script to use the docker image as a build agent where we run our integration tests:

RUN sudo apt-get -y install nodejs
RUN sudo apt-get install -y nodejs npm
RUN sudo npm install -g azurite
RUN sudo npm fund

To launch the included azurite instance, we issue the following command in our build pipeline's YAML file:

'sudo azurite --blobHost 0.0.0.0 --queueHost 0.0.0.0 --tableHost 0.0.0.0 -s -l /usr/local/lib/node_modules/azurite -d /usr/local/lib/node_modules/azurite/debug.log &'

We built the image and deployed to our k8s cluster a few months ago with the azurite version 3.29.0 and the integration tests have been working fine since then. We recently upgraded the Azure.Storage.Blobs ver 12.19.1 and Azure.Storage.Queues ver 12.17.1 NuGet packages that our integration tests use to versions 12.20.0 and 12.18.0, but the tests started failing after producing the following error messages despite we upgraded the azurite's instance to version 3.30.0:

2024-06-03T12:25:17.5563021Z 2024-06-03T12:25:17.5564076Z 2024-06-03T12:25:17.5564813Z 2024-06-03T12:25:17.5565576Z 2024-06-03T12:25:17.5566046Z 2024-06-03T12:25:17.5566595Z 2024-06-03T12:25:17.5567155Z 2024-06-03T12:25:17.5567782Z 2024-06-03T12:25:17.5568655Z 2024-06-03T12:25:17.5569359Z 2024-06-03T12:25:17.5569893Z 2024-06-03T12:25:17.5570383Z 2024-06-03T12:25:17.5570889Z 2024-06-03T12:25:17.5575045Z 2024-06-03T12:25:17.5576296Z 2024-06-03T12:25:17.5577262Z 2024-06-03T12:25:17.5577833Z 2024-06-03T12:25:17.5579070Z 2024-06-03T12:25:17.5579750Z 2024-06-03T12:25:17.5580282Z 2024-06-03T12:25:17.5580745Z Critical : ContainerWritingSASGenerator [0] System.AggregateException: Retry failed after 6 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (Connection refused (127.0.0.1:10000)) (Connection refused (127.0.0.1:10000)) (Connection refused (127.0.0.1:10000)) (Connection refused (127.0.0.1:10000)) (Connection refused (127.0.0.1:10000)) (Connection refused (127.0.0.1:10000))
---> Azure.RequestFailedException: Connection refused (127.0.0.1:10000)
---> System.Net.Http.HttpRequestException: Connection refused (127.0.0.1:10000)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) 2024-06-03T12:25:17.5571570Z at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) 2024-06-03T12:25:17.5572320Z at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) 2024-06-03T12:25:17.5572885Z at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async) 2024-06-03T12:25:17.5573574Z --- End of inner exception stack trace --- 2024-06-03T12:25:17.5574081Z at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async) 2024-06-03T12:25:17.5574560Z at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) 2024-06-03T12:25:17.5575678Z at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) 2024-06-03T12:25:17.5576785Z at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) 2024-06-03T12:25:17.5578349Z at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Storage.Blobs.ContainerRestClient.GetPropertiesAsync(Nullable`1 timeout, String leaseId, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobContainerClient.GetPropertiesInternal(BlobRequestConditions conditions, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobContainerClient.ExistsInternal(Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobContainerClient.ExistsAsync(CancellationToken cancellationToken)

Although, this failure does not happen on the development laptop where azurite ver 3.30.0 and same latest nuget packages are present.

Have you found a mitigation/solution?

No

The text was updated successfully, but these errors were encountered:

Arash-Sabet · 2024-06-03T16:42:38Z

@blueww Could you please chime in and let me know your thoughts on this issue? Thanks.

blueww · 2024-06-04T08:02:13Z

@Arash-Sabet

The above log you shared is from Storage SDK.
From the error I see: 1. the error happens on API BlobContainerClient.GetPropertiesInternal(); 2. the error is client can't connect to 127.0.0.1:10000 successfully.

Would you please:

Share the Azurite debug log when the failure happen? The debug log path is set with Azurite start parameter "-d /usr/local/lib/node_modules/azurite/debug.log" in your above Azurite start command. We need the Azurite debug log to do more investigation.
Share the connection string you use (hide credential), or just share do you use http or https?
Please check is there any config / process on your machine blocks the request send to 127.0.0.1:10000.

Arash-Sabet · 2024-06-04T12:47:17Z

@blueww

The debug log file was not created per the following screenshot, and it seems to be a new problem for you guys to solve:
The connection string is: "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;"
There is nothing blocking the request as the entire application used to work before upgrading the azurite and the nuget packages.

I also used --skipApiVersionCheck --loose in the azurite cli but it was not conclusive.

Arash-Sabet · 2024-06-06T10:56:42Z

@blueww any updates? Thanks.

blueww · 2024-06-07T10:34:15Z

@Arash-Sabet
We have never see any customer report azurite debug log not generated. So it’s almost not possible that there’s bug blocks azurite debug log generate.

Does the folder you show above on the docker , or on your client machine host docker? The debug log should be generated on the docker, so normally we will map the a folder on docker to a local folder, then generate debug log in the folder, then you can access debug log from local folder.

Another possible reason is azurite not start at all. There are something on your machine blocks azurite from starting.

Arash-Sabet · 2024-06-07T14:57:30Z

@blueww The following failure message is all I have to share. This exception also happens when we do not use sudo or with --loose or --skipApiVersionCheck switches.

sudo azurite --blobHost 0.0.0.0 --queueHost 0.0.0.0 --tableHost 0.0.0.0 -s -l /usr/local/lib/node_modules/azurite -d /usr/local/lib/node_modules/azurite/debug.log &
========================== Starting Command Output ===========================
/usr/bin/bash /azp/_work/_temp/5d5dabf2-5efa-43e2-974a-2ea32571caaf.sh
/usr/local/lib/node_modules/azurite/dist/src/common/persistence/MemoryExtentStore.js:53
return this._chunks.get(categoryName)?.chunks.get(id);
^

SyntaxError: Unexpected token '.'
at wrapSafe (internal/modules/cjs/loader.js:915:16)
at Module._compile (internal/modules/cjs/loader.js:963:27)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14)
at Module.require (internal/modules/cjs/loader.js:887:19)
at require (internal/modules/cjs/helpers.js:74:18)
at Object. (/usr/local/lib/node_modules/azurite/dist/src/common/ConfigurationBase.js:7:29)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)

blueww · 2024-06-11T08:59:10Z

@Arash-Sabet

From the above error, I really have no idea.
And I can't repro this.
I will try to contact the code owner of "MemoryExtentStore.js" to see if any idea.

One workaround is: you might can switch back to 3.29.0, if you work doesn't depends on change of 3.30.0.

Arash-Sabet · 2024-06-11T12:55:27Z

@blueww We have already had to downgrade the azurite to 3.29.0 and the NuGet packages too.

blueww · 2024-06-12T08:37:13Z

@Arash-Sabet

I get responds from the code owner of "MemoryExtentStore.js", however he seems also don't have much idea.
He mentioned:

I don't know. A syntax error may come from a very old version of node running on newer syntax .js file.
In memory persistence was introduced in 3.28.0. 3.29.0 works for the user but that also has in-memory persistence. Very confusing.

Arash-Sabet · 2024-06-12T13:04:41Z

@blueww That sounds like an issue in the implementation of azurite. This is a hinderance to us and other people who use azurite within docker. The issue has to be diligently resolved by Microsoft as the owner of this product. The response from the code owner is not satisfying as he must nail down this issue and address it. With this issue around, upgrading azurite NuGet packages becomes impossible unless you guys come up with another workaround.

blueww · 2024-06-14T03:57:53Z

@Arash-Sabet

The problem is the current error log is not enough to get the root cause of the issue.
We can't repro this issue, and we don't get other customer report similar issue.

We will need either a repro environment or more error log/trace to investigation this issue.
Could you give the very detail steps to repro this issue?
Or share a repro environment?

If we could get or setup a repro environment, it would definitely help the investigation.

BTW, could you check the node version of your repro environment?

Arash-Sabet · 2024-06-14T12:54:59Z

@blueww If other customers are not reporting this issue it does not mean that this problem does not exist. We cannot browse the entire internet to see what all people have to say or they may have not stumbled upon this issue yet. The error I shared is a fact from Azurite's error messages log that I collected.

I am unable to share an environment to reproduce this issue. But I hope the following docker file snippet leads your team to the issue:

FROM ubuntu:22.04
RUN DEBIAN_FRONTEND=noninteractive apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get upgrade -y

RUN DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \
    apt-transport-https \
    apt-utils \
    ca-certificates \
    curl \
    git \
    iputils-ping \
    jq \
    lsb-release \
    software-properties-common \
    sudo \
    zip \
    unzip \
    tar \
    wget

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

RUN wget https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
RUN dpkg -i packages-microsoft-prod.deb
RUN rm packages-microsoft-prod.deb
RUN sudo apt-get update
RUN sudo apt-get install -y dotnet-sdk-8.0

RUN sudo apt-get -y install nodejs
RUN sudo apt-get install -y nodejs npm
RUN sudo npm install -g azurite
RUN sudo npm fund
RUN sudo mkdir azurite

There are other commands that I excluded from this docker file deemed unnecessary in this scope. That's all I can provide.

blueww · 2024-06-17T10:49:12Z

@Arash-Sabet

For we don't get other customer report similar issue, I don't mean the issue not exist, but mean we don't have repro way from other customer, so can only get the way to repro it from you. And no other customer meet this issue, might will make the issue not so high priority compare with other features/issues requested by more customers.

We might can try to repro this with you above script. However, could you give the detail step by step guild on how to use the script to setup the docker, it will definitely make the repro more efficiently.

Arash-Sabet · 2024-06-17T12:54:10Z

@blueww I would give it a high priority to have it looked into.

To reproduce the issue: I'm sure there are other folks using k8s for their build agents. Please have your team follow the steps outlined per this blog post and include the script I shared when building the k8s's docker image. I guess that blog post is a re-iteration of a Microsoft's document/practice echoed by the author in the context of making a build agent.

Write a C# Xunit method to create a container on the Azurite instance, run it through a build pipeline, and you will observe the issue. This is how you'd normally launch Azurite inside the docker image from Azure DevOps' YAML file:

- task: Bash@3
  displayName: Launching Azurite
  inputs:
    targetType: 'inline'
    script: 'sudo azurite --blobHost 0.0.0.0 --queueHost 0.0.0.0 --tableHost 0.0.0.0  -s -l /usr/local/lib/node_modules/azurite -d /usr/local/lib/node_modules/azurite/debug.log &'

blueww · 2024-06-19T08:19:33Z

@Arash-Sabet

Thanks for the sharing!
Will look into this when finish the higher priority work in hand and update later.

blueww · 2024-07-03T07:33:09Z

@Arash-Sabet

I can get very similar error as yours when use a old version node (10.24.1) as following.
Azurite has already deprecated node 10/12. see #1830

Please try again with a later node version if you have not. I would suggest using LTS version v20.15.0. https://nodejs.org/en
If you have already used a supported Node version, please indicate which node version you are using.

D:\code\azurite\dist\src\common\persistence\MemoryExtentStore.js:53
return this._chunks.get(categoryName)?.chunks.get(id);
^

SyntaxError: Unexpected token .
at Module._compile (internal/modules/cjs/loader.js:723:23)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (D:\code\azurite\dist\src\common\ConfigurationBase.js:7:29)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)

Arash-Sabet · 2024-07-03T13:03:33Z

@blueww The following lines are from the script I already shared in this issue. Don't they imply that the latest node version is utilized?

RUN sudo apt-get -y install nodejs
RUN sudo apt-get install -y nodejs npm

blueww · 2024-07-08T03:13:02Z

@blueww The following lines are from the script I already shared in this issue. Don't they imply that the latest node version is utilized?
RUN sudo apt-get -y install nodejs
RUN sudo apt-get install -y nodejs npm

I am not sure.
A better way is get the node version after (or just before) the issue repro, and see if it use the correct node version.

Arash-Sabet · 2024-07-08T14:55:54Z

@blueww I am confused. Can you copy/paste your script that worked to see how you resolved the issue?

blueww · 2024-07-09T00:02:31Z

@blueww I am confused. Can you copy/paste your script that worked to see how you resolved the issue?

You can just run "node --version" to get the node version.

Arash-Sabet · 2024-07-09T16:17:55Z

@blueww that's not my point. My point is that what script you used to include that specific node version in your test script that led to success.

blueww · 2024-07-10T01:20:12Z

@blueww that's not my point. My point is that what script you used to include that specific node version in your test script that led to success.

For the setup script, this is out of Azurite scope, we might not be the best person to answer it. You might can contact related team like ADO team for how to setup the right node version.
Please make sure you are using right node version, and if so still any azurite issue happen, we can continue the investigation.

Arash-Sabet · 2024-07-10T12:23:08Z

@blueww The azurite clearly has a problem and to me the following script is downloading the latest version of node.

RUN sudo apt-get -y install nodejs
RUN sudo apt-get install -y nodejs npm

For the setup script, this is out of Azurite scope, we might not be the best person to answer it. You might can contact related team like ADO team for how to setup the right node version.

The setup script leads you to reproduce the problem and I don't think you cannot keep it out of this scope. This is a perfect test to reveal the problem in azurite. I would engage the team to solve the problem as you had asked me before to guide you through the reproduction steps. This is going to be my last answer and post on this issue.

blueww · 2024-07-11T08:31:42Z

@Arash-Sabet

Thanks for the reply!

However, Azurite can only work on the right node version. But how to setup the right node version on customer's machine is out of Azurite scope, customer owners the setup of node.

The error of this issue indicate it's very possible that a wrong node version is using, so to continue the investigation, please get the node version on your repro machine, and see if it's the correct node version.
For the installation script, we are not expert of node installation, so not sure if any configuration on client machine can impact the real node version be installed. The better way to check is: run "node --version" to get the real node version after the issue repro on the same machine.

Arash-Sabet · 2024-07-17T15:21:34Z

@bluewe Apparently we needed to get the docker file download and include the node's ver 20.x explicitly. Ver 20.x is not by default selected otherwise.

blueww self-assigned this Jun 4, 2024

blueww added question Further information is requested blob-storage labels Jun 4, 2024

Arash-Sabet closed this as completed Jul 17, 2024

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generating blob SAS tokens is failing with the NuGet package of version 12.20.0 and Azurite 3.30.0 #2406

Generating blob SAS tokens is failing with the NuGet package of version 12.20.0 and Azurite 3.30.0 #2406

Arash-Sabet commented Jun 3, 2024

Arash-Sabet commented Jun 3, 2024

blueww commented Jun 4, 2024 •

edited

Loading

Arash-Sabet commented Jun 4, 2024

Arash-Sabet commented Jun 6, 2024

blueww commented Jun 7, 2024

Arash-Sabet commented Jun 7, 2024

blueww commented Jun 11, 2024 •

edited

Loading

Arash-Sabet commented Jun 11, 2024

blueww commented Jun 12, 2024

Arash-Sabet commented Jun 12, 2024

blueww commented Jun 14, 2024 •

edited

Loading

Arash-Sabet commented Jun 14, 2024 •

edited

Loading

blueww commented Jun 17, 2024 •

edited

Loading

Arash-Sabet commented Jun 17, 2024 •

edited

Loading

blueww commented Jun 19, 2024 •

edited

Loading

blueww commented Jul 3, 2024 •

edited

Loading

Arash-Sabet commented Jul 3, 2024

blueww commented Jul 8, 2024 •

edited

Loading

Arash-Sabet commented Jul 8, 2024

blueww commented Jul 9, 2024

Arash-Sabet commented Jul 9, 2024

blueww commented Jul 10, 2024

Arash-Sabet commented Jul 10, 2024 •

edited

Loading

blueww commented Jul 11, 2024 •

edited

Loading

Arash-Sabet commented Jul 17, 2024

Generating blob SAS tokens is failing with the NuGet package of version 12.20.0 and Azurite 3.30.0 #2406

Generating blob SAS tokens is failing with the NuGet package of version 12.20.0 and Azurite 3.30.0 #2406

Comments

Arash-Sabet commented Jun 3, 2024

Which service(blob, file, queue, table) does this issue concern?

Which version of the Azurite was used?

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

What's the Node.js version?

What problem was encountered?

Steps to reproduce the issue?

Have you found a mitigation/solution?

Arash-Sabet commented Jun 3, 2024

blueww commented Jun 4, 2024 • edited Loading

Arash-Sabet commented Jun 4, 2024

Arash-Sabet commented Jun 6, 2024

blueww commented Jun 7, 2024

Arash-Sabet commented Jun 7, 2024

blueww commented Jun 11, 2024 • edited Loading

Arash-Sabet commented Jun 11, 2024

blueww commented Jun 12, 2024

Arash-Sabet commented Jun 12, 2024

blueww commented Jun 14, 2024 • edited Loading

Arash-Sabet commented Jun 14, 2024 • edited Loading

blueww commented Jun 17, 2024 • edited Loading

Arash-Sabet commented Jun 17, 2024 • edited Loading

blueww commented Jun 19, 2024 • edited Loading

blueww commented Jul 3, 2024 • edited Loading

Arash-Sabet commented Jul 3, 2024

blueww commented Jul 8, 2024 • edited Loading

Arash-Sabet commented Jul 8, 2024

blueww commented Jul 9, 2024

Arash-Sabet commented Jul 9, 2024

blueww commented Jul 10, 2024

Arash-Sabet commented Jul 10, 2024 • edited Loading

blueww commented Jul 11, 2024 • edited Loading

Arash-Sabet commented Jul 17, 2024

blueww commented Jun 4, 2024 •

edited

Loading

blueww commented Jun 11, 2024 •

edited

Loading

blueww commented Jun 14, 2024 •

edited

Loading

Arash-Sabet commented Jun 14, 2024 •

edited

Loading

blueww commented Jun 17, 2024 •

edited

Loading

Arash-Sabet commented Jun 17, 2024 •

edited

Loading

blueww commented Jun 19, 2024 •

edited

Loading

blueww commented Jul 3, 2024 •

edited

Loading

blueww commented Jul 8, 2024 •

edited

Loading

Arash-Sabet commented Jul 10, 2024 •

edited

Loading

blueww commented Jul 11, 2024 •

edited

Loading