Cache built-in tasks

[JamieMagee]

This repository contains the data for Azure Pipelines tasks from the marketplace, but the built-in tasks data is still contained in the main Renovate repository. The built-in task data should also be generated here.

• https://github.com/renovatebot/renovate/blob/main/tools/static-data/generate-azure-pipelines-tasks.mjs
• https://github.com/renovatebot/renovate/blob/main/data/azure-pipelines-tasks.json

[jessehouwing]

I have a much simpler script that accomplishes the same thing without parsing the whole git history now.

This API call on an instance in Brazil will be the first to provide a list of all available versions of all built-in tasks in a single call:

https://github.com/jessehouwing/azure-pipelines-tasks-zips/blob/b686589c370a4b15cde73d5fd1fd8e2bc538d020/download.ps1#L9

$org = "jessehouwing-brazil"
$pat = $env:AZURE_DEVOPS_PAT

$url = "https://dev.azure.com/$org"
$header = @{authorization = "Basic $([Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(".:$pat")))"}

$tasks = Invoke-RestMethod -Uri "$url/_apis/distributedtask/tasks?allversions=true" -Method Get -ContentType "application/json" -Headers $header | ConvertFrom-Json -AsHashtable

$taskMetadatas = $tasks.value

It would also be more accurate, because the git repo contains task versions that ultimately weren't rolled out to the public or were rolled out weeks later.

[viceice]

can you open a PR to it to this repo?

[jessehouwing]

@viceice @JamieMagee That turned out even simpler than I expected. Pull request opened. I'll be out of office the upcoming days.

[jessehouwing]

Ah, crap. Did a compare and it looks like Microsoft has cleaned up all old tasks that contain old vulnerable implementations. Those used to be available in every org. Must have been my complaints about the security of the marketplace that's been driving a cleanup act, I had been asking very pointed questions about this directly to a number of PMs.

So there are many old versions missing from the data generated this way. As far as I can tell never a version you'd want to move to, but it's not an historically complete dataset.

Interestingly the github data isn't "correct" either. it lists many task versions that never made it to the public, but is also somehow missing a number of versions that are returned from the "live" azure devops org.

Left is the git parsed file, Right is the API call generated file:

The 219 versions are missing from the left hand side and 2.220.1 seems to not have been released, it has probably been patched on the release/m220 branch prior to rollout and shipped as 2.220.2.

Here are the files both normalized in the same way:
diffable files.zip

[jessehouwing]

This is in the end why it's probably better to finish the work to read directly from the Azure DevOps Organization of the user, instead of trying to build a cache file:

renovatebot/renovate#19725

[viceice]

yes, but until someone finished that we should move the cache file here and out of renovate repo.

[jessehouwing]

Bluiltin tasks are now also updated.

[viceice]

so next step is to fetch the new file on the fly from renovate datasource like we do for the marketplace.