Docker use private NuGet Feed

[Hinton]

Environment

• VSTS

Issue Description

When building docker containers I would like to be able to use private NuGet feeds hosted on VSTS (Packaging). Currently there does not seem to be a good way to do this other than including a NuGet.config file with clear text credentials, which is less than optimal.

The .NET core task (v4) has options for NuGet feeds, and it would be nice if the Docker containers worked the same.

[bryanmacfarlane]

Are you building the container using 'docker build'?

[Hinton]

Yes, using the Dockerfile generated by Visual Studio, and the VSTS build template, Container (PREVIEW).

[bryanmacfarlane]

You can't use docker build for anything that has secrets (via files, envvars etc...) because the secret will end up being in the docker container you produce.

Instead, build the product including getting dev dependencies from nuget (that could also be in a container) to a mapped output container and then when you do docker build use a COPY statement in the docker build script.

We are working on first class docker support in VSTS which means you can say step 1 runs in container x and then run docker build.

If you email me at bryanmac (and that's at microsoft) I can share more details.

[Hinton]

I believe that's what my Dockerfile is currently doing, although it's using a multi-stage build.

FROM microsoft/aspnetcore:2.0 AS base
WORKDIR /app
EXPOSE 80

FROM microsoft/aspnetcore-build:2.0 AS build
WORKDIR /src
COPY *.sln ./
COPY NuGet.config ./
COPY Devices/Devices.csproj Devices/
RUN dotnet restore
COPY . .
WORKDIR /src/Devices
RUN dotnet build -c Release -o /app

FROM build AS publish
RUN dotnet publish -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "Devices.dll"]

Which is building correctly and the NuGet.config credentials are only accessible inside the build container which gets thrown away. I'm not particularly happy about including the credentials inside the repository but I guess that could be solved by creating the NuGet.config file inside the Agent using a shell script.

[bryanmacfarlane]

@Hinton - you're doing the dotnet restore in the docker file. You're also copying the nuget.config. What I'm saying is you do the dotnet restore and dotnet build outside the docker build file. The docker build file would copy in the resultant full product layout bits (your output).

[bryanmacfarlane]

... that also has the benefit of not requiring build tools in your products final container (small as possible). You can have a docker dev image that is based on the prod image and only the dev image has dev tools.

[Hinton]

@bryanmacfarlane I understand that's a potential way to solve it. Docker build would then only copy the pre-built binary files and not actually build anything.

However Visual Studio Docker Tools would then no longer be able to build and debug it through Visual Studio as a Docker target. Seems like Visual Studio Docker Tools still works without the build step.

The Dockerfile above is the recommended way to build an asp.net core 2.0 container. From my understanding of docker multi-step build, only the last FROM command is saved as the final container. The resulting container only contains the binary files based on the aspnetcore image (not aspnetcore-build).

[richlander]

@bryanmacfarlane Your proposal is antithetical to the promise of multi-stage build and to some degree to Docker itself. The promise of consistent and predictable execution environments applies nicely to build/CI environments as well.

On the question, I don't think we've got a great solution for this currently. Ideally, you could specify a nuget.config via an environment variable. I don't think this will work w/o getting NuGet to become secret aware. I'll ask.

[AceHack]

@richlander I agree with you. @bryanmacfarlane One of the big advantages of docker is being able to have a new developer build the solution with no prereqs installed except docker. In this way, they don't even need .NET core installed to build a .NET application. Also, @Hinton is correct in a multi-stage build only the last build stage is kept so, the build stage along with any secrets and/or SDK and build tools will not end up in the final docker image produced for your application.

[ghost]

@Hinton , @richlander, and @AceHack - I see where you're coming from. In VSTS we happen to have a focus on using containers for deployments. As such, we wanted to make sure you weren't putting VSTS PATs inside your deployments - and you're not! ✔️

It makes sense that you're asking for a better solution for the reproducible build scenario. We're chatting with @AndyGerlicher about this.

[AceHack]

@jerickmsft That's great to hear, I've worked up my own solution for passing though tokens to do authenticate with the registry. I'm now working on, unit tests and getting the results into VSTS, integration tests, code coverage, performance profiling, running static analysis, white source, and the list goes on and on. Doing your build inside the docker container makes many of the VSTS tasks hard or impossible to use. I'm having to come up with my own solutions to all of these. It's not really possible to find any guidance on these subjects either. Thanks.

[bozau]

Hi All,

I was wondering if there has been any progress on allowing a VSTS private NuGet package feed to be read from the dockerfile build process? I've tried creating a nuget.config file that can be then read by the dockerfile COPY nuget.config /root/.nuget/NuGet/ which is created at build time using a PAT token, but no success, still a 401 error.

Thanks.

[danilobreda]

What worked for me... using the PAT token.

My dockerfile:

FROM microsoft/aspnetcore-build:latest AS build
WORKDIR /src
COPY . .
COPY NuGet.Config ./
RUN dir
RUN dotnet restore --configfile NuGet.Config -nowarn:msb3202,nu1503 --verbosity diag
RUN dotnet publish --output /output --configuration Release

FROM microsoft/aspnetcore:latest
WORKDIR /app
COPY --from=build /output /app
ENTRYPOINT ["dotnet", "DockerProject.dll"]

and my NuGet.Config file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="BredasVSTS" value="https://xxxx.pkgs.visualstudio.com/_packaging/BredasVSTS/nuget/v3/index.json" />
  </packageSources>
  <packageSourceCredentials>
    <BredasVSTS>
      <add key="Username" value="emailhere" />      
      <add key="ClearTextPassword" value="PAT here" />
    </BredasVSTS>
  </packageSourceCredentials>
</configuration>

The Username parameter need to be passed but you can set a random username/phrase there... since PAT does not have username parameter.
Make sure that the nuget package have permissions (read and write) of PAT owner.

[bryanmacfarlane]

@danilobreda - is your PAT token now in a layer of your docker container?

[bryanmacfarlane]

@AceHack - regarding building in container and dependencies - agreed. Both solutions build in a container. Building in a container is great for dev == dev == ci. It's awesome.

Regarding tasks and the CI solution still working (tasks etc... ) and secrets not getting into layers (maps work dirs into container outside in), we will be previewing the ability to run any phase in a container (full yaml and task fidelty).

https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-phase.md#container-applies-to-queue

We're waiting on the rollout of a couple new hosted pools optimized for building containers. This and this.

I'm not advocating one over the other and docker build is an option. I'm just providing options while making it clear there's pros and cons to different approaches. Be aware of them. Pushing an image to a public repo with a token or cred in the layers is a real problem and concern.

We also have using kubernetes as a build pool (infra) in proof of concept phase working end to end.

[danilobreda]

@bryanmacfarlane

yes, it's because I'm not removing NuGet.Config, if I remove it, it will not be in any of the layers. Thank you for this note.

After some tests, its not on layers... the final docker is on the /output folder. It have only the "publish" output.

[bryanmacfarlane]

@danilobreda - one other note - does your current approach require you to check in your PAT to source code? (or some other process is generating it host side?)

[Hinton]

Since this issue is getting some activity again, I figured it might be worthwhile to describe, more in-depth how we worked around the issue (for now).

In our build definitions, before the docker tasks we have a bash task, which creates a NuGet.config file from a secret variable.

steps:
- bash: 'echo -e ''$(NuGet.config)' > NuGet.config' 
  displayName: Create NuGet.config

The secret variable is linked through a variable group to our built tasks and contains the complete NuGet.config file with the PAT.

This ensures the NuGet.config files with credentials are not stored in the source repositories. (And as previously discussed, the configs are not leaked to the final image)

Note that this works using the linux agents, on windows the bash script should be replaced with a PowerShell equivalent.

[felixdhdez]

For private nuget package or packages on Disk, I have the next Nuget.Config and Docker File. This works for me.

? xml version="1.0" encoding="utf-8" ?>
< configuration >
< packageSources >
< add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
< add key="AngryBirds" value="NugetPackage/" /> Local patch here
< /packageSources>
< packageRestore>
< add key="enabled" value="True" / >
< add key="automatic" value="True" / >
< /packageRestore>
< bindingRedirects>
< add key="skip" value="False" / >
< / bindingRedirects >
< packageManagement >
< add key="format" value="0" / >
< add key="disabled" value="False" / >
< /packageManagement>
< /configuration >

`FROM microsoft/aspnetcore:2.0 AS base
WORKDIR /app
EXPOSE 80

FROM microsoft/aspnetcore-build:2.0 AS build
WORKDIR /src

COPY NuGet.Config ./
COPY /NugetPackage/travelExpensesRestClient.1.0.0.nupkg NugetPackage/
RUN dir /src/NugetPackage

WORKDIR /src
##COPY %AppData%\NuGet ./
COPY travelExpenseLiquidador.sln ./
COPY travelExpense.Liquidador/travelExpense.Api.Liquidador.csproj travelExpense.Liquidador/
COPY travelExpense.Model.Peticiones/travelExpense.Model.Liquidador.csproj travelExpense.Model.Peticiones/
COPY travelExpense.DALC.Liquidador/travelExpense.DALC.Liquidador.csproj travelExpense.DALC.Liquidador/
COPY travelExpense.BO.Liquidador/travelExpense.BO.Liquidador.csproj travelExpense.BO.Liquidador/

#RUN dotnet restore -nowarn:msb3202,nu1503
RUN dotnet restore --configfile NuGet.Config -nowarn:msb3202,nu1503 #--verbosity diag
COPY . .
WORKDIR /src/travelExpense.Liquidador
RUN dotnet build -c Release -o /app

FROM build AS publish
RUN dotnet publish -c Release -o /app

#Try to create image from Publish output
#COPY / /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "travelExpense.Api.Liquidador.dll"]
`

[danilobreda]

@bryanmacfarlane ho, its on the code.. yes you are right.
I tried to use the Secure Files of the vsts, but i cant access it from the build container.

I think we can set the PAT as a enviroment variable... and pass it with the dockerfile ARG and ENV. Its something to test.

[bryanmacfarlane]

@danilobreda - yeah, that's what the yaml feature I pointed to above does. It gets source in the host and maps it into the container we spawn, then exec tasks in the container. That would allow you to use Secure Files and all other tasks. You also wouldn't need to copy it into the container or remember to delete. That needs a good dev == dev solution though.

Not sure, but I believe if you use ENV, that can get sealed in a layer. Might have to use ARG.

[cwoolum]

I have a Nuget custom task that adds the builds access token to the config. That way the token only lives for the life of the build and you don't have to worry about storing it in clear text in the config. Another nice thing is you don't need to remember to replace an expiring PAT.

sources Add -Name "MyPackages" -Source "https://my.pkgs.visualstudio.com/_packaging/MyPackages/nuget/v3/index.json" -username any -password $(System.AccessToken) -ConfigFile Source/Nuget.config -StorePasswordInClearText

Make sure that you also have"Allow scripts to access OAuth token" checked on the options tab.

I'm happy with this as a placeholder a better solution is presented.

[nphmuller]

Looks like this will be a reality soon-ish :)
docker/cli#1288

Just requires BuildKit to be enabled, which is possible from Docker 18.06-CE.

[cpuguy83]

"--secret" won't be supported on 18.06 daemons though, the version of buildkit
included with it doesn't have what is needed to make that work.

[nphmuller]

@cpuguy83 Sorry for the confusion. Meant that BuildKit's included (experimentally) since 18.06.

Looking at the PR --secret will likely be in 18.09, right?

[cpuguy83]

Correct.

[danilobreda]

So it will be possible to pass the PAT as a secret? How it will help for Docker use private NuGet Feed, can somebody clarify? Thanks.

[jerry-santana]

@cwoolum thank you for this snippet, it works.
Just a reminder for those who still get 401 even after applying @cwoolum solution, your PAT token needs to have Packages read permission included in their scopes.

Where can I configure the read permission? How can I know which PAT is being used?

[jerry-santana]

Seriously, when is this going to be fixed? I've done ALL the different solutions in this discussion and anything works! When is this going to be fixed? I've been blocked for the last 3 weeks.

[nkovacic]

@JerrySantana92 On Azure Devops click on your account and then security. There your token needs to have Agent pools (read & manage), Deployment Groups (read & manage) and Packaging (read).

PAT token is then passed as an argument as shown on screenshot.
$(System.AzureDevopsAccessToken) is a variable with your token from your account.

[camp-007]

The solution posted by @cwoolum seems to best solution I've encountered so far. Documentation on this scenario is VERY lacking. It seems like either the Azure Pipelines or Azure Artifacts docs should contain an example of the cross platform friendly way to do a package restore from a private Azure Artifacts feed from within a docker container. I wasted LOTS of time trying to get to this solution and we're still left wondering if this is the "best" or "recommended" approach (I suspect it is).

[jerry-santana]

@JerrySantana92 On Azure Devops click on your account and then security. There your token needs to have Agent pools (read & manage), Deployment Groups (read & manage) and Packaging (read).

PAT token is then passed as an argument as shown on screenshot.
$(System.AzureDevopsAccessToken) is a variable with your token from your account.

I've done that and still getting Unauthorized. Anything works for me.. this should be a bug in the task or pipeline.

[kristiansamuelsson]

The nice thing about this solution is that the ARG changing doesn't invalidate your hash so you can cache the entire dotnet restore step as long as your csproj files don't change.

@cwoolum Unfortunately, changing the value of an ARG does cause a cache miss on the first use of that variable, causing the rest of the image to be rebuilt (including the dotnet restore), see Dockerfile reference.

[cwoolum]

Using Docker build secrets, I was finally able to come up with a solution that properly handles caching and is also secure. Check out my blog post on it

https://medium.com/@cwoolum/using-nuget-restore-securely-inside-of-docker-containers-with-caching-2c2f5453905d

This solution doesn't work with Docker Compose yet and requires at least Docker 18.09 to work.

[nphmuller]

@cwoolum Really nice! The buildx plugin, which is available in the current 19.03 betas can also build compose files via BuildKit, which means it also has support for mounting secrets and caches. It works via the ‘docker buildx bake’ command.

Buildkit support in docker-compose depends on docker/docker-py#2230, which probably won’t be fixed in 19.03.

[nmehlei]

I may have misunderstood something, but none of these solutions seem to work for using Docker from Visual Studio with Windows-based containers, right?

apt-get would be out because of Linux, using Azure DevOps tasks is out because this wouldn't work inside Visual Studio and using Visual Studio project modifications like DockerfileBuildArguments don't seem to allow dynamic replacements like environment variables, only static text (which would commit the secret into the repo).

As of yet, I did not find a solution to use a .NET Core project with Windows-based Docker containers that allow running via F5 from inside Visual Studio. Am I missing something?

[cwoolum]

In Visual Studio, I've noticed that the build tools don't use the entire Dockerfile so restore works when building in Visual Studio. From what I can tell, it only uses the base image or the first few lines but I'm not positive on that. I've never had issues with F5 just working for Docker containers, no matter what Dockerfile I've had.

For making this work in a deployed environment, artifacts-credprovider does work on windows and you should be able to use the steps to install it to get it added as part of your image. From there, you should be able to use the same environment variable to authenticate. Just make sure that you are using a Windows Build host in Azure Pipelines so it can build Windows containers.

[heratyian]

I found this blog post really helpful and I take no credit for the solution. https://blog.ehn.nu/2019/05/accessing-azure-artifacts-feed-in-a-docker-build/

A workaround is to add a step in your dockerfile to download the microsoft azure cred provider and you pass your PAT as a build arg to docker.

docker build . --build-arg PAT=<pat-token>

# Dockerfile step
# Install Credential Provider and set env variables to enable Nuget restore with auth
ARG PAT
RUN wget -qO- https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh | bash
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS "{\"endpointCredentials\": [{\"endpoint\":\"<PRIVATE_ARTIFACT_FEED_HERE>\", \"password\":\"${PAT}\"}]}"

https://dev.azure.com/jakob/ignitetour/_git/DockerBuilds?path=%2F4.%20NugetRestore%2FWebApplication1%2FDockerfile&version=GBmaster

[osvaldolove]

The info provided by @heratyian works perfectly for me and is easy to implement.

[cwoolum]

We already have that solution above but it prevents the dotnet restore step from being cached by docker.

[osvaldolove]

@cwoolum I see you're other post about getting docker to cache with build secrets, I'll give that a whirl. There is a lot of info to parse in this thread!

[cwoolum]

I've come up with another way to use private Nuget repos in Azure Pipelines builds. It uses service containers so you'll need to be able to use YAML builds to support it.

The BuildKit solution I had above worked but you lost the ability to use --cache-from which is crucial to keeping build times down on Azure hosted agents. This new approach loads up a Nuget proxy in a container using your builds' access token to authorize and maps it to port 8080. From within your Dockerfile, you can restore using the host IP as a Nuget server. This eliminates the PAT arg which breaks the cache.

See the repo with the proxy and an example at https://github.com/cwoolum/ThinNugetProxy.

[pleongomez]

What worked for me... using the PAT token.

My dockerfile:

FROM microsoft/aspnetcore-build:latest AS build
WORKDIR /src
COPY . .
COPY NuGet.Config ./
RUN dir
RUN dotnet restore --configfile NuGet.Config -nowarn:msb3202,nu1503 --verbosity diag
RUN dotnet publish --output /output --configuration Release

FROM microsoft/aspnetcore:latest
WORKDIR /app
COPY --from=build /output /app
ENTRYPOINT ["dotnet", "DockerProject.dll"]

and my NuGet.Config file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="BredasVSTS" value="https://xxxx.pkgs.visualstudio.com/_packaging/BredasVSTS/nuget/v3/index.json" />
  </packageSources>
  <packageSourceCredentials>
    <BredasVSTS>
      <add key="Username" value="emailhere" />      
      <add key="ClearTextPassword" value="PAT here" />
    </BredasVSTS>
  </packageSourceCredentials>
</configuration>

The Username parameter need to be passed but you can set a random username/phrase there... since PAT does not have username parameter.
Make sure that the nuget package have permissions (read and write) of PAT owner.

Thanks! This worked for me. Regards!

[cwoolum]

This is normal. When you request your packages from the Azure Artifacts source, Azure Artifacts will look upstream at Nuget.org and cache the packages. You can remove Nuget.org as an upstream source but then you'll need to ensure your nuget config in your solution has that source and your Azure Artifacts source so it can search both.

…

On Tue, Mar 24, 2020 at 6:43 AM andrew-vdb ***@***.***> wrote: By doing this, it will push ALL dependencies to azure artifacts [image: image] <https://user-images.githubusercontent.com/12871379/77431837-8c768900-6ddd-11ea-99a4-f74bb09855cf.png> [image: image] <https://user-images.githubusercontent.com/12871379/77431913-aa43ee00-6ddd-11ea-93e3-cd4700e9c45d.png> Is this normal? expected? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6135 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHGCT6TT24JLTRG4QOT66TRJC2GJANCNFSM4EKHIBMA> .

[andrew-vdb]

@cwoolum Thanks for the reply, I already deleted my post... For anyone else, I was asking why the nuget.org packages being copied to my azure private nuget feed

Another problem now, for local development, regular container development mode, for build pipeline we use $(System.AccessToken)
What do we do with local development? at this moment I put my own PAT

I can't share this with others

Update:
I put my own PAT within this file now, .csproj.user

Is there any better way than this?

[cwoolum]

You wouldn't put that config in your project file. In Visual Studio 2019, you would already be authenticated via your account. For your builds, you would pass the PAT as a task parameter in your docker or docker-compose build step. See my posts above for more details. At no point should you need to specify the PAT manually.

…

On Tue, Mar 24, 2020 at 11:13 PM andrew-vdb ***@***.***> wrote: @cwoolum <https://github.com/cwoolum> Thanks for the reply, I already deleted my post... For anyone else, I was asking why the nuget.org packages being copied to my azure private nuget feed Another problem now, for *local development, regular container development mode*, for build pipeline we use $(System.AccessToken) What do we do with local development? at this moment I put my own PAT [image: image] <https://user-images.githubusercontent.com/12871379/77505326-1e74a500-6e63-11ea-9615-1dd5ae3ea048.png> I can't share this with others Update: I put my own PAT within this file now, .csproj.user [image: image] <https://user-images.githubusercontent.com/12871379/77506822-1a4a8680-6e67-11ea-9ee7-325320948013.png> Is there any better way than this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6135 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHGCT6IOQRMAC34M7MTKUTRJGOHZANCNFSM4EKHIBMA> .

[andrew-vdb]

In Visual Studio, I've noticed that the build tools don't use the entire Dockerfile so restore works when building in Visual Studio. From what I can tell, it only uses the base image or the first few lines but I'm not positive on that. I've never had issues with F5 just working for Docker containers, no matter what Dockerfile I've had.

I believe you use fast mode, not regular mode....

[richlander]

Have folks seen Managing NuGet Credentials in Docker Scenarios? It seems closely related to this topic.

/cc @mthalman

[florianeidner]

It might be the same approach, but I prefer to use the dotnet cli to add the nuget source + credentials. Especially if you use a multi-stage build, the nuget config that stores the token is removed afterwards. There is no need to add a specific packages as i think. Especially if you run it Azure Pipelines you can just inject the $(System.AccessToken) as the PAT argument. It would look like this in the dockerfile

ARG PAT
COPY . .
RUN dotnet nuget add source "your-source-url" --name "source-name" --username "useless" --password "$PAT" --store-password-in-clear-text
RUN dotnet restore

[edgarrs]

As @richlander pointed out this is something to follow up on the dotnet-docker repo:
https://github.com/dotnet/dotnet-docker/blob/master/documentation/scenarios/nuget-credentials.md

[vikramadhav]

Reading through so many solution and implementing them , i was huffed and puffed .

please don't forget to add username in ENV variable otherwise you will have 401.

ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS "{\"endpointCredentials\": [{\"endpoint\":\"https://something.pkgs.visualstudio.com/_packaging/Nuget/nuget/v3/index.json\", \"username\":\"build\", \"password\":\"${PAT}\"}]}"

[GF-Huang]

My question is: How to make PAT safe when I also use docker-compose.yml to build the Dockerfile.

[jakehockey10]

I'm trying this latest approach and the Docker build step isn't interpolating the variable usage for the build-args:

I tried the above with no quotes and single quotes as well.

Here is the important part of my Dockerfile:

And when I look at the logs of the failed build, I see that it is sending the password to my private source like this:

[johnterickson]

Reminder: Do NOT NOT NOT put secrets/credentials as ENV vars into layers that you ship. Docker loves to store your environment variables as part of the layer. If you are not careful you will ship your credentials with the layers you create.

See:
https://github.com/dotnet/dotnet-docker/blob/master/documentation/scenarios/nuget-credentials.md
https://medium.com/@cwoolum/using-nuget-restore-securely-inside-of-docker-containers-with-caching-2c2f5453905d

[madibaT]

My question is: How to make PAT safe when I also use docker-compose.yml to build the Dockerfile.

did you come right with this?