Fix the broken template scanning pipeline job #7724
Conversation
| @@ -0,0 +1,24 @@ | |||
| { | |||
There was a problem hiding this comment.
I added this file to make f5 debugging of the tool as it is run in the pipeline quite simple to do.
| @@ -11,6 +11,7 @@ namespace Microsoft.TemplateSearch.Common | |||
| /// <summary> | |||
| /// Template package searchable data. | |||
| /// </summary> | |||
| [System.Diagnostics.DebuggerDisplay("{Name}@{Version}")] | |||
There was a problem hiding this comment.
This workflow deals with large lists of these items, having a debugger visualization is very handy.
| @@ -6,6 +6,7 @@ | |||
|
|
|||
| namespace Microsoft.TemplateSearch.TemplateDiscovery.PackChecking | |||
| { | |||
| [System.Diagnostics.DebuggerDisplay("{Name}@{Version} - {Reason}")] | |||
There was a problem hiding this comment.
This workflow deals with large lists of these items, having a debugger visualization is very handy.
| throw new Exception($"temp storage path for NuGet packages already exists: {_packageTempPath}"); | ||
| } | ||
| else | ||
| if (!Directory.Exists(_packageTempPath)) |
There was a problem hiding this comment.
These two changes allow us to re-use the already-downloaded nupkgs when re-running the application. This really cuts down on cycle time.
tools/Microsoft.TemplateSearch.TemplateDiscovery/PackChecking/PackSourceChecker.cs
Show resolved
Hide resolved
|
I'm doing a quick compare between my local version of the search cache and the version available at the search cache url - will report back if the diffs seem sensible |
|
Converted this to a draft because there were a lot of removals in the new cache generated locally, and I'm not confident in the results quite yet. |
tools/Microsoft.TemplateSearch.TemplateDiscovery/NuGet/NugetPackProvider.cs
Outdated
Show resolved
Hide resolved
|
Ok, I took the time to do a detailed diff and now I'm happy with the results. There's some shuffling of what templates are/are not in the cache file, but these is just a side-effect of the way we are querying the overall NuGet API - because we're stuck at the first 4kish responses due to the API we're using and there's no consistent package ordering, we're going to vary somewhat on the packages in the overall listing. |
| @@ -12,12 +14,14 @@ internal partial class TemplateSearchCache | |||
|
|
|||
| internal TemplateSearchCache(IReadOnlyList<TemplatePackageSearchData> data) | |||
| { | |||
| TemplatePackages = data; | |||
| // when creating from freshly-generated data, order the results for clarity | |||
There was a problem hiding this comment.
This change should make it easier for us to do side-by-side compares in the future. When I did this, it made it very easy to see that most diffs were like this:
meaning descriptions, versions, download counts that had changed over time. The other big source of diffs was additions/removals of entire packages from the cache file, which is inevitable until we move to searching the entire NuGet feed.
|
Two runs are failing, but I can't see the results - are they not reported in AzDO? |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Ok, a rerun fixed this. Since it's been approved and I'm confident it in, going to merge it now. We should check the pipeline runs in an hour or so and see if they green up. |
Problem
The pipelines for the template cache scanner have been broken for some time now due to an invariant changing - the tools never expected that a template would just...be completely removed.
Solution
We allow this case and add it to the set of removed templates. It can always be re-added back once it's back on NuGet.org.