Feature: Index Schema Field Filtering

[michael-valdron]

Please specify the area for this PR

What does does this PR do / why we need it:

Adds field-based filtering of index schemas with fuzzy search and and logical based filter stacking, e.g. filter by name and description.

Additional Changes:

• Added deprecated filter parameter and filter process: REST API query parameters for requesting field-based filtering of index schemas api#959 (comment)
• Endpoint unit tests now use generated ServerInterfaceWrapper rather than explicitly defined Server that implements generated ServerInterface to better test parameter wrapper along with greatly increasing code coverage, see endpoint_test.go#L481 for an example of this change
• Use of github.com/mohae/deepcopy in filter to produce a different copy of the filtered index schema
• Use of github.com/hashicorp/go-set to use sets where it is more efficient
• Use of k8s.io/apiextensions-apiserver to set field pointers easier
• Introduction of new helper functions
  • StructToMap to transform a struct type into a map[string]any
  • StrPtrIsSet to return the result of ptr != nil && *ptr != "" for string pointers
• Introduction of IndexParams and toIndexParams receiver functions to generalize index endpoint parameter types from generated source
• Removal of GET method parameters from not allowed method handlers
• Correct typos under endpoint unit tests

Which issue(s) this PR fixes:

Fixes #?

fixes devfile/api#959

PR acceptance criteria:

• Test Coverage
  • Are your changes sufficiently tested, and are any applicable test cases added or updated to cover your changes?

Documentation (WIP)

• Does the REST API doc need to be updated with your changes?
• Does the registry library doc need to be updated with your changes?

How to test changes / Special notes to the reviewer:

1. Build devfile-index image from change
2. Deploy built devfile-index image
3. Build & run altered integration test suite on deployment of built devfile-index from step 1
4. Run cd index/server && go test ./... to run unit tests on changes

[codecov]

Codecov Report

Attention: Patch coverage is 54.76540% with 617 lines in your changes are missing coverage. Please review.

Project coverage is 41.16%. Comparing base (7c89891) to head (5b6f1cf).
Report is 1 commits behind head on main.

Files	Patch %	Lines
index/server/pkg/server/endpoint.gen.go	23.53%	380 Missing and 91 partials ⚠️
index/server/pkg/util/filter.go	76.62%	110 Missing and 5 partials ⚠️
index/server/pkg/server/endpoint.go	75.00%	15 Missing and 4 partials ⚠️
index/server/pkg/util/util.go	77.14%	5 Missing and 3 partials ⚠️
index/server/pkg/server/filter.go	86.20%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #202       +/-   ##
===========================================
+ Coverage   27.27%   41.16%   +13.88%     
===========================================
  Files           7        9        +2     
  Lines        2009     3102     +1093     
===========================================
+ Hits          548     1277      +729     
- Misses       1414     1636      +222     
- Partials       47      189      +142     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[thepetk]

@michael-valdron I've tried to build locally and test the new devfile-index image with minikube. When I'm trying to get filtered results I get the full list of stacks. What I'm trying to fetch is:

http://devfile-registry-default.<IP>.nip.io/v2index?name=python

Any ideas on what I'm doing wrong?

The dockerfile I'm using to build the image is:

FROM registry.access.redhat.com/ubi8/go-toolset:1.19 AS builder

# Set user as root
USER root

# Install yq
RUN curl -sL -O https://github.com/mikefarah/yq/releases/download/v4.9.5/yq_linux_amd64 -o /usr/local/bin/yq && mv ./yq_linux_amd64 /usr/local/bin/yq && chmod +x /usr/local/bin/yq

COPY . /registry

# Download the registry build tools
RUN git clone --branch feat/index-filtering https://github.com/michael-valdron/devfile-registry-support.git /registry-support

# Run the registry build tools
RUN bash /registry-support/build-tools/build.sh /registry /build

# Set user as non-root
USER 1001

FROM quay.io/devfile/devfile-index-base:next

COPY --from=builder /build /registry

[michael-valdron]

@michael-valdron I've tried to build locally and test the new devfile-index image with minikube. When I'm trying to get filtered results I get the full list of stacks. What I'm trying to fetch is:

http://devfile-registry-default.<IP>.nip.io/v2index?name=python

@thepetk I wasn't able to reproduce exactly what you got (all stacks), I got an empty result.

The empty result would be because this endpoint /v2index (and /v2index/stack) only queries stacks whereas the only python entry in the test registry is a sample, you can pull the desired result with:

curl http://devfile-registry-default.<IP>.nip.io/v2index/sample?name=python

You can filter this against stacks and samples with:

curl http://devfile-registry-default.<IP>.nip.io/v2index/all?name=python

[michael-valdron]

When I'm trying to get filtered results I get the full list of stacks.

I wasn't able to reproduce exactly what you got (all stacks)

@thepetk What steps did you do to build and deploy? Maybe there is something else missed.

[thepetk]

curl http://devfile-registry-default.<IP>.nip.io/v2index/sample?name=python

I'll re-do again all steps just to avoid the case that the devfile-index image wasn't up-to-date

[thepetk]

@michael-valdron as we've checked together (thanks for the help debugging my issue!), I'd suggest adding a custom field for deprecated stacks.

This way when someone needs to fetch a deprecated stack (a stack with the Deprecated tag inside the default tags) will fetch only the fully deprecated stacks (without a supported version).

e.g

https://<host>/v2index/all?deprecated=true

[thepetk]

@michael-valdron I've tried to build locally and test the new devfile-index image with minikube. When I'm trying to get filtered results I get the full list of stacks. What I'm trying to fetch is:

http://devfile-registry-default.<IP>.nip.io/v2index?name=python

Any ideas on what I'm doing wrong?

The dockerfile I'm using to build the image is:

FROM registry.access.redhat.com/ubi8/go-toolset:1.19 AS builder

# Set user as root
USER root

# Install yq
RUN curl -sL -O https://github.com/mikefarah/yq/releases/download/v4.9.5/yq_linux_amd64 -o /usr/local/bin/yq && mv ./yq_linux_amd64 /usr/local/bin/yq && chmod +x /usr/local/bin/yq

COPY . /registry

# Download the registry build tools
RUN git clone --branch feat/index-filtering https://github.com/michael-valdron/devfile-registry-support.git /registry-support

# Run the registry build tools
RUN bash /registry-support/build-tools/build.sh /registry /build

# Set user as non-root
USER 1001

FROM quay.io/devfile/devfile-index-base:next

COPY --from=builder /build /registry

For everyone's information the issue here was the quay.io/devfile/devfile-index-base:next. In order to be able to properly test the changes I had to build the base image from the base branch.

[Jdubrick]

Can't seem to build and run the integration tests because my podman alias is not being properly set so I will need to look into this before I can run the tests on my end. I was able to build and deploy the devfile-index however and seemed to work with no issues.

[michael-valdron]

Can't seem to build and run the integration tests because my podman alias is not being properly set so I will need to look into this before I can run the tests on my end. I was able to build and deploy the devfile-index however and seemed to work with no issues.

Opened devfile/api#1502 to address this along with devfile/api#1503 to discover any other missing/broken podman support in our registry-support scripts.

FYI @Jdubrick @thepetk

[michael-valdron]

@michael-valdron as we've checked together (thanks for the help debugging my issue!), I'd suggest adding a custom field for deprecated stacks.

This way when someone needs to fetch a deprecated stack (a stack with the Deprecated tag inside the default tags) will fetch only the fully deprecated stacks (without a supported version).

e.g

https://<host>/v2index/all?deprecated=true

I have included this additional parameter with the some criteria: devfile/api#959 (comment)

FYI @thepetk @Jdubrick

[thepetk]

/lgtm tested locally all the filters & the additional deprecated=true/false :)

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: michael-valdron, thepetk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [michael-valdron,thepetk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment