Sync NPM data to graph

[msrb]

Description

Not all NPM data made it all the way to graph. We need to sync them.

Acceptance criteria

• all NPM data from production S3 buckets are available in graph

[sivaavkd]

@tuxdna - are we targeting this for the current sprint ? cc @msrb

[msrb]

@sivaavkd yeah, the plan is to have the NPM data in graph by the end of the sprint. The sync process is already running and @tuxdna is keeping an eye on it. Also there is a parallel effort to make the sync faster.

[tuxdna]

The graph sync ran for few days now, and we have 92695 NPM package versions in the pending list. There are no more package-versions going through the sync anymore.

There are known reasons due to which this is happening

• package data (prod-bayesian-core-package-data) for a particular version is not present in S3 but package version does (prod-bayesian-core-data) have data in S3. For example there are 75 versions for arachne-ui but there is no package data present.
• there is no data at all in S3 for a particular package
• other reasons we haven't figured out yet.

Essentially for NPM, the graph-sync is done to the extent which it could be done in the current state.

[tuxdna]

Here is the summary for package version counts using BookKeeping and GraphSync APIs

From: GET /bookkeeping

{
  "summary": [
    {
      "name": "rubygems",
      "package_count": 0,
      "package_version_count": 0
    },
    {
      "name": "npm",
      "package_count": 175658,
      "package_version_count": 416920
    },
    {
      "name": "maven",
      "package_count": 178756,
      "package_version_count": 635159
    },
    {
      "name": "pypi",
      "package_count": 104166,
      "package_version_count": 273733
    },
    {
      "name": "go",
      "package_count": 84825,
      "package_version_count": 102239
    },
    {
      "name": "crates",
      "package_count": 0,
      "package_version_count": 0
    },
    {
      "name": "nuget",
      "package_count": 94114,
      "package_version_count": 199632
    }
  ]
}

From: GET /graphsync/pending/npm

{"data": {
    "all_counts": 92715,
    "pending_list": [{ ... }]}}

From above we have 416920 package version count for NPM in total, and out of these 92715 are not synced to graph yet because of reasons mentioned in earlier comment above. This gives us 77.76 % coverage of NPM data by package-version counts in Graph:

>>> (1 - 92715 / 416920) * 100
77.76192075218266

[msrb]

@tuxdna

package data (prod-bayesian-core-package-data) for a particular version is not present in S3 but package version does (prod-bayesian-core-data) have data in S3. For example there are 75 versions for arachne-ui but there is no package data present

I don't think that having no package-level data should be an issue for graph import. In case of arachne-ui, it looks like an error in ingestion pipeline, but in general, there can be components for which we simply won't have any package-level data.

Could you please check in data-importer, how it behaves when there are no package-level data? And file an issue, if it fails :) Thanks 😉

[tuxdna]

@msrb: Sure, will check and update.