feat(third-party): add handling of Angular CLI schematics, and rework registry subset

[Haroenv]

This way we don't change the keywords anymore, and can do actual tests on the package contents.

cc @manekinekko 🙌

@vvo I forgot which name you suggested instead of registry subset

closes #167

[vvo]

Why not just ...getKeywords(cleaned) to avoid the __ and temp variable

[vvo]

If we only have an array of keywords for now and not actual metadata then let's just call it "computedKeywords".

[Haroenv]

I'm not sure about the name keywords

[Haroenv]

technically they are more like identifiers than keywords

[vvo]

Subset is how we see one usecase for this feature today. If someone reads the registry it must be clear what those are, so I guess computedKeywords is what this data is: a list of keywords we compute

[manekinekko]

Very nice. So, If I get this right, regarding the Angular use case, and given the two following packages (for instance):

(*) these are Angular Schematics

A. @nrwl/bazel (*)

{
  "name": "@nrwl/bazel",
  "description": "Nrwl Extensions for Angular: Schematics",
  "keywords": [
    "angular",
    "schematics"
  ],
  "schematics": "./src/collection.json",
  ...
}
// Note the presence of the "schematics" custom metadata.

B. @angular/pwa (*)

{
  "name": "@angular/pwa",
  "description": "PWA schematics for Angular",
  "keywords": [
    "angular",
    "devkit",
    "tooling",
    "schematics"
  ],
  "schematics": "./collection.json",
  ...
}
// Note the presence of the "schematics" custom metadata.

C. @angular-devkit/schematics

{
  "name": "@angular-devkit/schematics",
  "description": "Angular Schematics - Library",
  "keywords": [
    "angular",
    "devkit",
    "tooling",
    "schematics"
  ],
  ...
}
// Note the lack of the "schematics" custom metadata.

(*) these are Angular Schematics

Searching for packages ...

1. ... matching angular will return packages A-B-C.
2. ... that are Anguar Schematics, will return A-B.
3. ... that are Anguar Schematics, and containing bazel will return A.
4. ... that are Anguar Schematics, and containing pwa will return B.
5. ... that are Anguar Schematics, and containing devkit will return B.

Are the above valid use cases?

[Haroenv]

How can we have the last one, but not accidentally every package that has something to do with an unrelated kind of schematics? I only looked at the package.json key right now.

[vvo]

Can you summarise what this PR changes in term of the response schema? Right now if I get it well it does adds a new property containing computed keywords. But seems like we want to go further and be able to add:

{
  "name": "algoliasearch",
  "computedKeywords": ["angular", "schematics"], // computed by npm-search, not original data, used for faceting/filtering
  "computedMetadata": { // used for displaying
    "schematics": "/some/path.json"
  }
}

Is that correct and what we want? ^

[vvo]

@manekinekko what's the search UI you are going to provide? It can help seeing it as for usecases and filtering.

If you filter on angular-schematics to only get this subset, all your usecase should work. @Haroenv Not sure to get why last one would not work.

[manekinekko]

@vvo here is a video of a working demo: https://twitter.com/manekinekko/status/978027847804751873

And here is the code: manekinekko/angular-cli@6efa6c3

As you can see from this line, I'm simply looking for keywords. Basically searching for any NPM packages that contains angular+schematics+${query}. But this is not ideal since we can get many false positives. The best way when it comes to look for Angular Schematics, is to filter out the pakages that only have the schematics custom metadata in their package.json.

I have an open RFC here: angular/angular-cli#10000

[Haroenv]

what's done in this PR will make a subset from the index to only have the packages which have the key schematics, but your example c wouldn't have that. Which kind of pattern do you think would match that too, without false positives like the keyword would have?

[vvo]

Then "having the shematics metadata" must be translated into an actionnable filter, like:

{
  "name": "algoliasearch",
  "computedKeywords": ["angular", "schematics"], // computed by npm-search, not original data, used for faceting/filtering
  "computedMetadata": { // used for displaying
    "hasSchematics": true,
    "schematics": "/some/path.json"
  }
}

Because there's no filter in Algolia that checks for attribute presence or not. Wdyt?

[vvo]

LGTM

[manekinekko]

I'm not familiar with actionable filters, but that sounds great ;)

How about making it more generic like:

"computedMetadata": { // used for displaying
    "has": ["schematics"],
  }

So It would also allow searching for packages that contain any custom metadata, like a jest entry, or foobar:

"computedMetadata": { // used for displaying
    "has": ["jest", "foobar"],
  }

Or better,

"computedMetadata": { // used for displaying
    "has": [
      {
         "jest": {
             "verbose": true
         }
      }, 
      "foobar"
    ],
  }

This would only allow limit the search to packages that contains either a jest entry with a verbose:true props, or packages that contain a foobar entry.

Or maybe this is out of scope?

[Haroenv]

All you'll need to do this instead of your usage of search earlier:

index.search({ query, facetFilters: ['computedKeywords:angular-cli-schematic']})

(it's a Promise, so no need for a callback unless you really want) @manekinekko

[Haroenv]

The actual schematics will be in computedMetadata.schematic by the way (but it's not completely reindexed yet)