[Catalog] AND filter don't work for same field

[angeliski]

Expected Behavior

When I use the filter for tags, I should receive only entities with react and javascript tag, like the docs

URL: /api/catalog/entities?filter=metadata.tags=react,metadata.tags=javascript

In my example (entities below), should receive only app1

Actual Behavior

I am trying to filter entities with react and javascript tag, but the results are entities with react or javascript tag
URL: /api/catalog/entities?filter=metadata.tags=react,metadata.tags=javascript

code:

const result = await catalogClient.getEntities({
                filter: [
                    {
                        'metadata.tags': ['react', 'javascript']
                    }
                ]
            },
            {token}
        )

Entities in my catalog:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: app1
  description: app1 service
  tags:
    - javascript
    - react
    - app1
spec:
  type: service
  owner: devtools
  lifecycle: production
---
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: app2
  description: app2 service
  tags:
    - javascript
    - app2
spec:
  type: service
  owner: devtools
  lifecycle: production
---
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: app3
  description: app3 service
  tags:
    - react
    - app3
spec:
  type: service
  owner: devtools
  lifecycle: production

Result example (using field selection for simplicity)

Steps to Reproduce

1. Load entities above in the catalog
2. Put in your browser the url http://localhost:7007/api/catalog/entities?filter=metadata.tags=react,metadata.tags=javascript
3. You will receive the wrong entities (3 entities in this case)

Investigation

Looking for the current implementation in the catalog, this behavior is expected.
In the DefaultEntitiesCatalog we translate repetead values to a in clausule (here) so the result is an or opeation

Context

We are building some features using the tags entities as reference so we don't need to extend the catalog and add complexity.
Example: We want to perform some operations (like an codebase update) in javascript and react entities, but not in javascript and vue apps. But the filter isn't working, so we will need to get all javascript and filter after that.

Your Environment

• Browser Information: Google Chrome Versão 105.0.5195.102 (Versão oficial) 64 bits

• Output of yarn backstage-cli info:

yarn run v1.22.18
$ /home/ROGERIO.ANGELISKI/workspace/backstage/node_modules/.bin/backstage-cli info
OS:   Linux 5.14.0-1051-oem - linux/x64
node: v16.14.0
yarn: 1.22.18
cli:  0.18.1 (installed)
backstage:  1.5.1

Dependencies:
  @backstage/app-defaults                           1.0.5
  @backstage/backend-common                         0.15.0
  @backstage/backend-plugin-api                     0.1.1
  @backstage/backend-tasks                          0.3.4
  @backstage/catalog-client                         1.0.4
  @backstage/catalog-model                          1.1.0
  @backstage/cli-common                             0.1.9
  @backstage/cli                                    0.18.1
  @backstage/config-loader                          1.1.3
  @backstage/config                                 1.0.1
  @backstage/core-app-api                           1.0.5
  @backstage/core-components                        0.10.0, 0.11.0
  @backstage/core-plugin-api                        1.0.5
  @backstage/errors                                 1.1.0
  @backstage/integration-react                      1.1.3
  @backstage/integration                            1.3.0
  @backstage/plugin-analytics-module-ga             0.1.19
  @backstage/plugin-api-docs                        0.8.8
  @backstage/plugin-app-backend                     0.3.35
  @backstage/plugin-auth-backend                    0.15.1
  @backstage/plugin-auth-node                       0.2.4
  @backstage/plugin-catalog-backend-module-github   0.1.6
  @backstage/plugin-catalog-backend                 1.3.1
  @backstage/plugin-catalog-common                  1.0.5
  @backstage/plugin-catalog-graph                   0.2.20
  @backstage/plugin-catalog-import                  0.8.11
  @backstage/plugin-catalog-node                    1.0.1
  @backstage/plugin-catalog-react                   1.1.3
  @backstage/plugin-catalog                         1.5.0
  @backstage/plugin-circleci                        0.3.8
  @backstage/plugin-github-actions                  0.5.8
  @backstage/plugin-github-issues                   0.1.0
  @backstage/plugin-github-pull-requests-board      0.1.2
  @backstage/plugin-home                            0.4.24
  @backstage/plugin-org                             0.5.8
  @backstage/plugin-pagerduty                       0.5.1
  @backstage/plugin-permission-common               0.6.3
  @backstage/plugin-permission-node                 0.6.4
  @backstage/plugin-permission-react                0.4.4
  @backstage/plugin-proxy-backend                   0.2.29
  @backstage/plugin-rollbar-backend                 0.1.32
  @backstage/plugin-rollbar                         0.4.8
  @backstage/plugin-scaffolder-backend-module-rails 0.4.3
  @backstage/plugin-scaffolder-backend              1.5.1
  @backstage/plugin-scaffolder-common               1.1.2
  @backstage/plugin-scaffolder                      1.5.0
  @backstage/plugin-search-backend-node             1.0.1
  @backstage/plugin-search-backend                  1.0.1
  @backstage/plugin-search-common                   1.0.0
  @backstage/plugin-search-react                    1.0.1
  @backstage/plugin-search                          1.0.1
  @backstage/plugin-stack-overflow                  0.1.4
  @backstage/plugin-tech-radar                      0.5.15
  @backstage/plugin-techdocs-backend                1.2.1
  @backstage/plugin-techdocs-module-addons-contrib  1.0.3
  @backstage/plugin-techdocs-node                   1.3.0
  @backstage/plugin-techdocs-react                  1.0.3
  @backstage/plugin-techdocs                        1.3.1
  @backstage/plugin-user-settings                   0.4.7
  @backstage/release-manifests                      0.0.5
  @backstage/test-utils                             1.1.3
  @backstage/theme                                  0.2.16
  @backstage/types                                  1.0.0
  @backstage/version-bridge                         1.0.1
Done in 0.66s.

[freben]

Yeah this is actually by design. The documentation doesn't explicitly point out this corner case, but the comment in the code does:

backstage/packages/catalog-client/src/types/api.ts

Line 38 in 5b2d037

/**

Within one key, if there are more than one value, then there is effectively an OR between them.

Sorry but this is the way the current filter sets are designed. We've been talking about whether there could be reason to design a complete query language where you can use combinators as needed, but no solid plans yet.

[angeliski]

Make sense @freben

Maybe we could update the docs to put this info?

Do you have any suggestion to achieve this same result (a AND operation for equal fields)

[freben]

I don't think so, no. You could make two separate filter sets (one that matches on a, one that matches on b) with all of the other conditions duplicated. That way you'll get all entities that have one, the other, or both. And then post filtering in code.

Or keeping it simple and filtering only on one of the labels if you know that it's much less common of the two and, again, post filtering in code.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.