Accessing Git repo

[pascalberger]

This is maybe not directly on issue with the tasks but more with the agent. What is the recommended approach to run Git operation during a build?

My use case is that I have a repository utilizing git-lfs. Since VSTS only does a checkout I manually need to call git lfs fetch afterwards. The problem here is the authentication. I want to avoid to have to setup an service account for accessing the repo just for this case. The agent already somehow manages to checkout the repo. Is it possible to use the same approach for additional operations?

[bryanmacfarlane]

The current prepare does it via libgit and uses the generated per build OAuth token sent down to the agent.

That oauth token is available to custom task authors so that's one possibility.

However in the sprint rolling out right now (S92 - in a couple scale units so far), we added a checkbox on the options tab to make the token available to 'ad-hoc scripts'. This means you trust everyone that can edits your definition (and all process and scripts that run it**) :) It's a variable (can't remember off the top of my head - do a set and inspect)

Now, onto getting git.exe to use it. There's ASKPASS and credhelpers but those are actually problematic with per build passwords since cred helpers can store and then you don't get called back, then your token for that build expires, next build fails :(

So, one option is to edit your remote. Call get to set the remote to Ouath:@ At the end of the script you can set it back.

We are contemplating doing what I just mentioned for you as an option. If we do that (and you check that box), ad-hoc git scripting via cmd files just works. What won't work is authenticated sub-modules with that approach since the url is embedded in the sub-module ref

[pascalberger]

Thanks. I will wait for S92 and try the OAuth token / edit remote way.

[pascalberger]

@bryanmacfarlane I tried to update the git remote using the SYSTEM_ACCESSTOKEN environment variables but couldn't make it work. Did I get it right that I should be able to replace the url of the origin remote (https://mytenant.visualstudio.com/DefaultCollection/MyProject/_git/MyRepo) with the OAuth token like this: https://<Token>@mytenant.visualstudio.com/DefaultCollection/MyProject/_git/MyRepo) and then be able to perform git lfs operations? Unfortunately I still get an "Git credentials not found error"

[bryanmacfarlane]

Needs to be username:password

So use any username like OAuth and then token as password

[pascalberger]

Thanks, https://OAuth:<Token>@mytenant.visualstudio.com/DefaultCollection/MyProject/_git/MyRepo unfortunately still doesn't work

[bryanmacfarlane]

Hmmm - that's what our xplat agent does. I'll try it out later. Until then any other details like the cmds run, output, scripts etc would help

[pascalberger]

I've written a build task which updates the remote. You can find it in this PR or in this ZIP.

My test build definition is:

• The custom build task
• A command line task calling git with lfs fetch as argument

Allow Scripts to Access OAuth Token is set in the options. Repository clean is set to true.

The repo contains one binary file which was added with git lfs.

While calling the git lfs fetch it throws the Git credentials for https://mytenant.visualstudio.com/DefaultCollection/MyProject/_git/MyRepo/info/lfs/objects/841e3a4174433932fc9e0a7f506cfda7f240b640199a656632300a971d11bd58 not found. error.

If I manually call git remote -v in the Git repository on the agent after running the build it shows the remotes being updated with username/password.

[bryanmacfarlane]

Just as a test - does it work if it's just gig fetch? Trying to figure out if it's an lfs thing ...

[pascalberger]

A git fetch works fine. Seems to be an lfs issue

[pascalberger]

@bryanmacfarlane I opened a ticket on git-lfs regarding this: git-lfs/git-lfs/issues/906

[pascalberger]

Closing the issue since the VSTS related part seems to work fine with standard Git commands. Lfs related stuff should be handled in git-lfs/git-lfs/issues/906.

Thanks @bryanmacfarlane for the help!

[rmarinho]

As some feedback i think we could have a GetSources as a task,where we have a primary source but can i have other sources.. TeamCity allows this with multiple VCS roots

[ericsciple]

@rmarinho agreed. we are working toward that.

[ggirard07]

@ericsciple is there any reference on VSTS features timeline about it I can look for to know when this will be completed and actually released?

[ericsciple]

@ggirard07 no but we've been wanting to do it for a long time and we're working a feature now that i think is going to force it (next few months would be my best guess). we've also been making incremental steps along the way to decouple get sources from the agent.

for a workaround today you could clone the repo yourself. if you check Allow scripts access to OAuth token then you can use the token for the Project Collection Build Service identity.

[fahnzmode]

@ericsciple - would you happen to have any additional info on the possibility of VSTS builds being able to checkout multiple git repositories? Would be very nice to have first-class support for this.

Dave Burnison mentions the steps needed to cleanly clone the git repo yourself:

1. Use a PowerShell task to call a script that:
   – Changes directories to the $(build.sourcesDirectory)
   – Creates a new folder called “Build” under $(build.sourcesDirectory)
   – Changes directories to the $(build.sourcesDirectory)\Build
   – Calls “git init”
   – Calls “git clone https://VSTSAccountName.visualstudio.com/SomeCollection/Build/_git/BuildUtilsStoredInGit” where the URL points to a Git repository that has a set of scripts that are similar to $/Build/PSScripts in the blog post.
2. Use additional PowerShell tasks to execute the script(s) as needed from $(build.sourcesDirectory)\Build (e.g. $(build.sourcesDirectory)\Build\PSScripts\ApplyVersionToAssemblies.ps1 )
3. At the end of your build process add a command line task to call the following command which will remove the local copy of the Git repository
   – RmDir /S /Q “$(build.sourcesDirectory)\Build”

(see his 'June 1, 2016 at 8:22 pm' comment here: https://blogs.msdn.microsoft.com/premier_developer/2016/04/13/tips-for-writing-powershell-scripts-to-use-in-build-and-release-tasks/)

[ericsciple]

It's something we want to do for multiple reasons. It's not next though.