REST API query parameters for requesting field-based filtering of index schemas

[michael-valdron]

Which area this feature is related to?

/area registry

Which functionality do you think we should add?

Why is this needed? Is your feature request related to a problem?

Allows a consumer of the REST API to request for filtered down responses when working with GET /index and GET /v2index.

Detailed description:

Currently for GET /index and GET /v2index, there are a few query parameters which provide filtering based on schema version and architecture type. In addition to these, it would be good to have the following querying capabilities:

• Stack/Sample query (fuzzy search) filter using index schema fields excluding select one
• New lastModified filter

These capabilities would reduce load on both the registry index server and REST API consumers such as the registry viewer.

Describe the solution you'd like

Tasks

• Stack/Sample query parameters for filtering by index schema properties
  • Filtering using AND
• Stack/Sample lastModified named query parameter See REST API lastModified query parameter for requesting filtered index schemas #1327
• Additional testing additions to account for changes done
• Additional documentation additions to REST API docs to describe these parameters

Examples

Request: GET /index?name=python or GET /v2index?name=python

Response:

[
  {
    "name": "python",
    "displayName": "Flask",
    "description": "Flask is a web framework, it’s a Python module that lets you develop web applications easily. It’s has a small and easy-to-extend core: it’s a microframework that doesn’t include an ORM (Object Relational Manager) or such features.",
    "type": "stack",
    "tags": [
      "Flask",
      "Python",
      "Pip"
    ],
    ...
  },
  {
    "name": "python-django",
    "displayName": "Django",
    "description": "Django is a high-level Python web framework that enables rapid development of secure and maintainable websites. Built by experienced developers, Django takes care of much of the hassle of web development, so you can focus on writing your app without needing to reinvent the wheel. It is free and open source, has a thriving and active community, great documentation, and many options for free and paid-for support.",
    "type": "stack",
    "tags": [
      "Django",
      "Python",
      "Pip"
    ],
    ...
  }
]

Describe alternatives you've considered

Without these changes, the filters in this issue would need to be applied on the client side which both requires the registry index server to create the entire list of index schemas to send as the response and put additional load on the consumer having to filter and pull out needed information after the fact.

Additional context

[elsony]

1. For the fuzzy search, what fields are we using to do the matching? Do we use name+description+tags or some other combination?
2. Do we need a search based on the schema version (or combination of schema version)?

[schultzp2020]

@elsony

1. For the fuzzy search, what fields are we using to do the matching? Do we use name+description+tags or some other combination?

That works for me.

2. Do we need a search based on the schema version (or combination of schema version)?

We already have a search based on the schema version

[michael-valdron]

@elsony @schultzp2020

1. For the fuzzy search, what fields are we using to do the matching? Do we use name+description+tags or some other combination?

That works for me.

I will update the description to include this.

[schultzp2020]

@michael-valdron Could we have a filter for all the logical filterable devfile properties?

What I am thinking... (You already pointed out a few of these in the pr)

• name
• type? Maybe we should change the API so it defaults to returning all the stacks/samples and then you need to specify ?type=<stacks || samples>
• tags
• projectType
• language
• provider
• schemaVersion? Change the way this is done so its ?schemaVersion=2.1 or ?schemaVersion=2.1-2.2 where schema version accepts a SEMVER-like version or a range

Also, for the different versions, its so minimal I can filter it out easily on the frontend. So its up to you if you think its necessary to add. If so, instead of it being a query parameter can you add a property called availableVersions that contains the array of version numbers?

Lastly, is it possible to add a parameter called lastModified that returns the last modified date for the corresponding devfile? I already discussed this feature request with @elsony and he likes it. If so, can you make this queryable? This would help the registry viewer.

[michael-valdron]

@schultzp2020

• tags
• projectType
• language
• provider

Lastly, is it possible to add a parameter called lastModified that returns the last modified date for the corresponding devfile? I already discussed this feature request with @elsony and he likes it. If so, can you make this queryable? This would help the registry viewer.

These I am good with including.

• type? Maybe we should change the API so it defaults to returning all the stacks/samples and then you need to specify ?type=<stacks || samples>
• schemaVersion? Change the way this is done so its ?schemaVersion=2.1 or ?schemaVersion=2.1-2.2 where schema version accepts a SEMVER-like version or a range

It might be complicated as changing these might break functionality with current consumers.

Also, for the different versions, its so minimal I can filter it out easily on the frontend. So its up to you if you think its necessary to add. If so, instead of it being a query parameter can you add a property called availableVersions that contains the array of version numbers?

Not sure what you mean by adding a property called availableVersions. Do you mean instead of doing something like GET /index?versionList=true might be better to do GET /index/availableVersions or GET /index/available-versions?

[schultzp2020]

@michael-valdron

Not sure what you mean by adding a property called availableVersions. Do you mean instead of doing something like GET /index?versionList=true might be better to do GET /index/availableVersions or GET /index/available-versions?

I mean we do something like this. Adding this index/functionality isn't necessary for the registry viewer though.
GET /index

[
  {
     ...
    "language": "go",
    "provider": "Red Hat",
    "availableVersions": ["1.1.0", "1.2.0"],
    "versions": [
      {
        "version": "1.1.0",
        "schemaVersion": "2.0.0",
        "default": true,
        ...
      },
      {
        "version": "1.2.0",
        "schemaVersion": "2.1.0",
        "description": "Stack with the latest Go version with devfile v2.1.0 schema verison",
        "tags": [
          "testtag"
        ],
        "icon": "https://raw.githubusercontent.com/devfile-samples/devfile-stack-icons/main/golang.svg",
        ...
      }
    ]
  },
...
]

[michael-valdron]

@schultzp2020

Not sure what you mean by adding a property called availableVersions. Do you mean instead of doing something like GET /index?versionList=true might be better to do GET /index/availableVersions or GET /index/available-versions?

I mean we do something like this. Adding this index/functionality isn't necessary for the registry viewer though. GET /index

[
  {
     ...
    "language": "go",
    "provider": "Red Hat",
    "availableVersions": ["1.1.0", "1.2.0"],
    "versions": [
      {
        "version": "1.1.0",
        "schemaVersion": "2.0.0",
        "default": true,
        ...
      },
      {
        "version": "1.2.0",
        "schemaVersion": "2.1.0",
        "description": "Stack with the latest Go version with devfile v2.1.0 schema verison",
        "tags": [
          "testtag"
        ],
        "icon": "https://raw.githubusercontent.com/devfile-samples/devfile-stack-icons/main/golang.svg",
        ...
      }
    ]
  },
...
]

I don't see why we can't have this addition to the V2 index schema. I can do this instead of having a ?versionList query parameter to fetch this list from versions.

[michael-valdron]

@schultzp2020 @elsony

Was there a preferred datetime format for lastModified? Should we just use the date?

[schultzp2020]

@michael-valdron ISO 8601 YYYY-MM-DD

[michael-valdron]

Based on recent changes to the description, this issue is no longer a dependency for #950.

[schultzp2020]

@michael-valdron Can you add a query to show x number of devfiles or a range of devfiles?

e.g. GET /index?show=10 would show the first 10 devfiles and GET /index?show=11-20 would show the 11th through 20th devfiles.

[schultzp2020]

@michael-valdron Also, I would look into https://github.com/google/go-querystring because I am using https://www.npmjs.com/package/query-string on the frontend with the array format as comma.

[michael-valdron]

As discussed in the community slack channel:

Decided form of the filters in the Registry REST APIs /index and /v2index query strings will be in the form ?field1=val1&field2=val2 along with reserved operational query parameters: e.g. show, minSchemaVersion, and maxSchemaVersion.