feat(changelog): get from jsDelivr filelist if possible

[bodinsamuel]

getChangelogs is currently the slowest operation and probably the one with the most impact on the network, because it queries N files in parallel multipled by X packages.

To help this: try to fetch file list from jsDelivr to check Changelogs presence.
If it matches we should gain 4s avg (up to 8s), if it does not matches (can happen that changelog are not published along with the code) we rollback to the brut-force method.

[Haroenv]

looks clear now! maybe some tests where there's "almost-right" files in jsDelivr that shouldn't be marked as resolved so that doesn't regress?

[bodinsamuel]

maybe some tests where there's "almost-right" files in jsDelivr that shouldn't be marked as resolved so that doesn't regress?

Not sure I got that, can you explain?
edit: I have added a test should not register a file looking like a changelog with a file changelog.js to avoid regression, if that's what you are talking about?

[bodinsamuel]

The call is not as fast as I hoped, I mean it's seems fast in the browser but it's probably because it's cached.
Considering most of the calls are very specific, I think we are never hitting the cache.

The average is ~4s.

Maybe @jimaek has some input on this?

the function is here:

npm-search/src/jsDelivr/index.ts

Lines 77 to 102 in fbe2e97

	export async function getFilesList(
	pkg: Pick<RawPkg, 'name' | 'version'>
	): Promise<File[]> {
	const start = Date.now();
	if (!pkg.name || !pkg.version) {
	throw new Error(
	`Package name should contain a version number: ${pkg.name}`
	);
	}
	let files: File[] = [];
	try {
	const response = await request<{ default: string; files: File[] }>(
	`${config.jsDelivrPackageEndpoint}/${pkg.name}@${pkg.version}/flat`,
	{
	responseType: 'json',
	}
	);
	files = response.body.files;
	} catch (e) {
	log.warn(e);
	}
	datadog.timing('jsdelivr.getFilesList', Date.now() - start);
	return files;
	}

[MartinKolarik]

@bodinsamuel if you request listings for every package, that seems about right (for now). There are two cache layers:

1. The CDN - if cached there, the response time should be ~10 ms. This would require the exact URL match at the exact geographic location.
2. Our internal database - if stored there, the response time should be < 100 ms. This database is global and persistent so after the first request, the listing should always be stored here.

However, if we don't have the listing stored yet (the very first request for that package/version), the response time is in seconds. I'm assuming that if you call this for every single package, you're going to end up with lots of DB misses at the beginning.

[bodinsamuel]

Makes sense, and since we will probably re-request at each version increment we will almost always cache miss ahah.
we are building the cache for you 😬 (hope it's okay)

Thanks for the answer

[Haroenv]

when we re-request the same package, won't it be for a different version, and thus a cache miss? I'm not sure what the alternative is though, maybe unpkg can be an option, if it has a higher cache rate? or requesting and untarring in advance to make the process non-blocking?

[jimaek]

unpkg will have the same issue, it will have to download the npm and cache it. So first request for a new version will also be a miss and pretty slow.
How big of a problem is this latency? Maybe we could figure something out to help you? We have our own users using the same API so any cache misses are basically cache warming.

[bodinsamuel]

How big of a problem is this latency?

It's not a problem per say, I understand the "issue" of cache miss.
I replaced our naive polling of every possible URLs to find readmes/changelog to use jsdelivr /flat API. I expected to gain a lot of time, but because we cache miss a lot, it's not as effective as I hoped.
And because we do that for 1.7million NPM packages it's normal that you don't have the cache for every single one of them. However the average is very constant toward 4s (but decreasing).

(nb: We are fetching alphabetically and currently in the packages beginning by @types/)

[jimaek]

@MartinKolarik maybe we could somehow speed things up on our side?

[MartinKolarik]

Looking at our DB, we have > 2.5M package versions cached (1.2M unique packages) but there might be some other things slowing this down. At least in some cases, I see that the npm API that we use for resolving versions was unusually slow to respond and took most of the overall processing time.

I'm going to check this further but in the meantime I'd also ask you to add a User-Agent header (e.g., Algolia npm-search (https://github.com/algolia/npm-search))to your requests as per our API recommendations to make analyzing this easier.

[bodinsamuel]

Good point, adding it in #646

[MartinKolarik]

@bodinsamuel we didn't make any changes yet but I see an average response time of 1300 ms for the past 24 hours for your requests. Do you see a similar improvement in your metrics?

[Haroenv]

The average indeed went down a bit:

[MartinKolarik]

I deployed an optimized version at around 5 PM UTC, hopefully, you'll see some additional improvement.

[bodinsamuel]

Thanks for taking some time to look at that Martin 🙏🏻
The average did not move but we are now more often around 2s than at the start

[MartinKolarik]

Interesting, this is how it looks from our end (specifically for your requests):

[bodinsamuel]

we don't observe that sharp drop unfortunately, that's weird.
maybe there is more to it (DNS resolution, Time to download, etc...) but that is a big diff 😦

Unrelated to that I have noticed a few other things:

• 500 on some packages, e.g: https://data.jsdelivr.com/v1/package/npm/gital@0.0.0/flat
• Sometimes a just published version is reported as not existing in your backend, e.g: https://data.jsdelivr.com/v1/package/npm/cra-template-react-json-api-server-kumard3 (it exists now)

404

Just published in npm

---

• We had a few spikes when reaching jsdelivr, but not sure it's from your backend. Could be network congestion.

[MartinKolarik]

500 on some packages, e.g: https://data.jsdelivr.com/v1/package/npm/gital@0.0.0/flat

That one is corrupt, try to download and open it: https://registry.npmjs.org/gital/-/gital-0.0.0.tgz

Sometimes a just published version is reported as not existing in your backend, e.g:

That may indeed happen sometimes if you request it too soon due to the various caches involved.

Btw, I'm also seeing an increase in request throughput, which corresponds with the per-request latency drop.

[bodinsamuel]

Yes I have changed our infra and parallelism to process way more packages.
We definitely see a decrease since we started using jsdelivr but not as low as what you shown, not sure which numbers to trust ahah. Honestly I don't know if there is anything to do

last 4hours, average between 1-2s with big spike at 30s

last week trend