4+ days indexing delay

[MartinKolarik]

Based on #1110, #1111, and jsdelivr/www.jsdelivr.com#518 it seems there's currently more than 4 days of delay in the indexing process.

[hoijnet]

Looks like I have the same issues, currently 10 days and counting for this package now: `@dfrnt/gatsby-source-graphql-nodes, also another package that I know people have started using successfully in Gatsby that is also not added yet gatsby-mdx-remote.

[hoijnet]

Still no update?

[Haroenv]

Sorry, there's not much we can do to make this faster, the processing is quite complex and easily gets overloaded.

[hoijnet]

I take it an update is on its way, but it has been indexing for about two weeks now? Fingers crossed it completes soon ☀️

[MartinKolarik]

Not sure if there was an issue with just one package or if the indexing is broken entirely now, but https://www.jsdelivr.com/package/npm/await-sync is missing from the index and the first version was published more than a month ago: "1.0.0": "2023-04-19T12:37:10.779Z", "1.0.1": "2023-05-08T18:32:55.625Z"

[Chaitanya-Raj]

The same issue is occurring with https://www.jsdelivr.com/package/npm/@teamepyc/sliders. Loading the files from the CDN works but it doesn't show up in the search.

[zachmerrill]

My company's NPM package version in search hasn't indexed since March 21st.

@Haroenv any update on this? The CI failed 4 months ago and your Datadog shows no data. Is the indexing up and running?

[arcanis]

The problem likely comes from npm, but no progress on their side yet: https://github.com/npm/feedback/discussions/937 😕

[Eprince-hub]

I am also having some issues with the indexing, as it seems that the @apollo/experimental-nextjs-app-support package has not been indexed since it was released a few months ago on npm.

[arcanis]

It seems some of the missing packages have been synced to replicate.npmjs.org, but they are still missing from the Algolia search. For instance Vite is still referencing 4.2 as the last version, and patch-package as 6.x.

[MartinKolarik]

We're now working together with Algolia to improve the sync process. Will post when I know more about the current status.

[quinton-ashley]

How have you guys not fixed an issue of this magnitude that's been known about since April? It's been 4 months.

[vishwas-r]

The issue is same with https://www.jsdelivr.com/package/npm/@canvasjs/charts package also! It's been more than 3 months since the package was published, but not getting indexed & shown in the search result yet!

[jasongodev]

I just encountered this nonsense and I guarantee a PostgreSQL implementation of indexing jsdelivr and npm will be much faster in updating. Basic full text search will work decently in npm's search context where keywords and a description forms the basic search context.

Why are we using Algolia aside from "hey it's Algolia, failure resilient search, shiny object cool gizmo because it's Algolia, sounds regal", except that it has not updated for 5 months! That's failure and that's not resilient.

Maybe it's time to reevaluate and rethink the search feature and indexing for npm packages.

[quinton-ashley]

@jasongodev Don't worry I made a PR #1139 that fixes the issue. :)

[MartinKolarik]

@jasongodev the current state of this project is a result of multiple problems combined but none of them is related to the actual search engine. A PostgreSQL or any other implementation would have the same problems.

That said, we (jsDelivr) had a productive call with the Algolia team last week and I think all current issues can be resolved during the next month.

[quinton-ashley]

@MartinKolarik that's great news

[MartinKolarik]

We're now running a fresh bootstrap with many improvements (#1140). It'll take about a week to reindex everything and if it's all good after that, the new index will likely replace the current one.

[MartinKolarik]

As discussed in #1140, we expect all changes (and all up-to-date data) to be live tomorrow.

[hoijnet]

Fantastic! Finally a solution in sight! Looking forward tonseeing the results of your work!!!

[karlhorky]

Thanks so much @MartinKolarik and reviewers @pixelastic and @bodinsamuel for the considerable effort in #1140 🙌

This is going to affect a lot of downstream users!

[MartinKolarik]

Everything is live now! Let us know if you find any issues.

[termosa]

@MartinKolarik I was following this thread hoping that it will fix an issue for me.

But CodeSandbox still doesn't show the use-request even if the whole name is written in the search.
I have been talking with their support and they have said that they cannot do anything as it is on side of Algolia.

Any hope that it can be resolved somehow?
Thank you in advance!

[MartinKolarik]

@termosa if you check here the module is listed (in the 6th place) so it is present in the Algolia data.

[termosa]

Terrific. I surmise that the ranking does not depend solely on the name. Therefore, even if a query exactly matches the name, it doesn't necessarily guarantee that it will be the first one.
Alright, I can proceed from here.
Thank you very much, @MartinKolarik!