Migration path for the change to require a repo resource for versions repo

[dagood]

When I saw this failure in the microsoft/go-images Publish stage when trying to upgrade:

/mnt/vss/_work/_temp/289fe1ea-058b-44fa-a948-b0fee38b4f65.sh: line 1: cd: /mnt/vss/_work/1/s/versions: No such file or directory

I tracked down:

• Use service connection to pull latest image info during publish #1744

Slight breaking change: All publishing pipelines which publish image info now need to specify a repo resource. That's the cost of avoiding rate limiting.

I started the process to mirror the Go versions repo into dnceng. But I think I misunderstood, and it's actually expected that the repo resource continues to point directly at the source of truth repo (GitHub) for synchronization purposes.

Is it true that we must point at the repo on GitHub, and set up whatever authorization is necessary to get that done?

Or: a line of questions come to mind, but maybe I'm missing something:

• What happens if we do use the mirror, and there's mirror latency? (Might some amount of latency be acceptable?)
• In what circumstances must the data be 100% up to date? (Maybe microsoft/go-images is simple enough that it doesn't actually matter for us in particular?)

---

And an alternative: could we use gitHubVersionsRepoInfo.authArgs to use the GitHub API to read/update this file, rather than using both git and the app? We already use it to publish, could it also help with reading the latest?

(Or: another app with less permission.)

Maybe this depends too much on the access being in dnceng/internal, while the repo reference is also needed in dnceng-public in some cases. For Go, we forgo the caching features in public CI, which I think means we don't use it but I'm not sure if this is the only case.

[lbussell]

What happens if we do use the mirror, and there's mirror latency?

The GitHub -> dnceng/internal mirror can exhibit up to 15 minutes of latency. The problem is if you have two image publishing pipelines going at the same time, and one finishes and publishes new images before the other, and the other pipeline isn't able to pull the latest image-info due to the latency, then it won't be able to push changes to GitHub due to having a stale reference. That's also why we try to pull the latest changes from GitHub at the last possible moment during publishing.

In what circumstances must the data be 100% up to date? (Maybe microsoft/go-images is simple enough that it doesn't actually matter for us in particular?)

If you don't think Golang is susceptible to the above issue, then maybe an internal mirror can work. If you have multiple build pipelines pushing separate files to the same versions repo, though - it can be a problem.

could we use gitHubVersionsRepoInfo.authArgs to use the GitHub API to read/update this file, rather than using both git and the app?

I am open to changes that allow you to "bring your own" versions repo and auth. The reason we did the repo clone this way is because it seemed like the "correct"/SFI compliant way to do it with Azure pipelines. But if there's a way to let individual repos make that choice then I'm on board with that.

---

One interesting thought experiment, which is kind of tangential to this issue - I wonder if we can reverse the direction of the mirror. Maybe we could always use an AzDO repo for builds, and rely on some other pipeline/service to push updates to GitHub. That'd get rid of this problem entirely.

[lbussell]

Related:

• Use GitHub instance of versions repo as the sole source of image info data #986
• During publishing, "Download Latest Image Info" step can fail with 429 too many requests #1737 (comment)

[dagood]

One interesting thought experiment, which is kind of tangential to this issue - I wonder if we can reverse the direction of the mirror. Maybe we could always use an AzDO repo for builds, and rely on some other pipeline/service to push updates to GitHub. That'd get rid of this problem entirely.

If we can get permission to push directly to a branch in AzDO this sounds quite good, and with the existing GitHub app we could even update the GitHub repo ourselves immediately rather than relying on a polling operation later. I worry it's not feasible to get the ability (in AzDO) to push directly though...

If you have multiple build pipelines pushing separate files to the same versions repo, though - it can be a problem.

Is this a fundamental problem, or just something that hasn't been implemented? It seems like separate files should be ok in theory--if the reference is stale, essentially rebase until it works or it turns out the same file's been edited--but I can see how it wouldn't work if not specifically accounted for.

We do have two repos/files, but I think we can tolerate scheduling them carefully and even occasional failures, so I think the mirror might work for us. (Can always change later, we already have the mirror anyway.)

The reason we did the repo clone this way is because it seemed like the "correct"/SFI compliant way to do it with Azure pipelines.

It makes sense, my mind just went immediately to internal mirror--I suppose I have lingering allergies to direct GitHub access. 😄 With this repo really just being a database, it's a decently unique case.

I could see using a GitHub app potentially seeming similar to reimplementing a Git service connection and rejecting that approach along those lines. I would potentially argue that because a git clone isn't necessary (maybe a clone is actually worse, security-wise, compared to reading individual files?), using the GitHub API makes sense.

I can see how this might be more complicated in dnceng-public--I've only had "simple" CI pipelines there so far.

[lbussell]

[Triage] @dagood is this blocking your adoption of the latest eng/common changes? Or is your workaround with the internal mirror working for now?

[dagood]

It doesn't seem like it'll be a blocker, but had to pause the migration for some unrelated work. I hope to know more sometime this week. (Last error was some 1ES PT stuff caused by having the new repo involved that I expect I'll be able to sort out.)

[dagood]

It seems to work, I got a test build to be almost entirely green (just an SDL Sources job path fail waiting for a PR to that repo). We'll have to see over the coming weeks/months if we hit any other issues that are too problematic.

I noticed one thing I worked around differently than you, but I didn't end up having to modify eng/common to make it work for us, so no blocker here:

docker-tools/eng/common/templates/steps/init-matrix-build-publish.yml

Lines 69 to 78 in fc11659

	- ${{ if ne(parameters.versionsRepoRef, '') }}:
	# Special logic is needed to copy the tsaoptions.json file to a well known location for the 1ES PT.
	# This template has multiple checkouts and AzDO doesn't have support for dynamically determining the
	# default repo path therefore the 1es-official logic can't calculate the repo's tsa config file path.
	- task: CopyFiles@2
	displayName: Copy TSA Config
	inputs:
	SourceFolder: '$(Build.Repository.LocalPath)/$(buildRepoName)'
	Contents: '.config/tsaoptions.json'
	TargetFolder: '$(Build.SourcesDirectory)'

The problem is that we also have a .config/CredScanSuppressions.json file. (Some ordinary Go stuff ends up in the spdx and appears to CredScan to be a secret (sometimes).) For now I have this in 1ES PT:

      credscan:
        suppressionsFile: $(SDLConfigFileDir)/CredScanSuppressions.json
      tsa:
        enabled: true
        configFile: $(SDLConfigFileDir)/tsa/tsaoptions.json

These variables:

- name: SDLConfigFileDir
  value: $(Build.Repository.LocalPath)/.config
- name: SDLConfigFileMultiRepoDir
  value: $(Build.Repository.LocalPath)/microsoft-go-images/.config

And this customPublishInitStep:

              - pwsh: echo "##vso[task.setvariable variable=SDLConfigFileDir]$(SDLConfigFileMultiRepoDir)"
                displayName: 'Set SDLConfigFileDir for multi-checkout'

I did this before I noticed the tsaoptions.json-specific workaround... I probably would have changed eng/common if I'd seen it, but this variable swap seems to work fine for now, and I don't see any big reason to go one way or another in the future. 😄

[dagood]

Tentatively closing as answered: main migration path is to get a service connection set up, a second is to set up an internal mirror (with known caveats).

I think doing this check/update with a GitHub App could make sense, but it can be considered on its own merits, if it's even worth that time. Reversing the mirror also sounds like a good approach, but seems to me like it would have more security surface area to look at. (At least for our teams.)