Unable to use Flux OCI with Harbor robot accounts

[antgamdia]

Summary
I've been adding minor changes for the UI to also support the OCI registries with Flux (PR #5218), in doing so, I'm kind of unable to add one (harbor) and get the packages.

Background and rationale

It's important as in many Harbor deployments we don't have full admin access, but maybe admin permissions over a single project (not the whole harbor instance). Using robot accounts seems to be the preferred way to consume TAC/VAC, therefore, I guess it's important that we also support this way to enable VAC users to deploy their apps using kubeapps+flux.

Description

In the logs I see:

ORAS .Repositories() returned err: GET "https://harbor-repo.vmware.com/v2/_catalog?last=agamez": unexpected status code 401: unauthorized: unauthorized to list catalog: unauthorized to list catalog

I assume that this is because the robot accounts are not privileged enough (see goharbor/harbor#8723 (comment)).

However, the following endpoint seems to work with robot accounts and does give me the list of artifacts inside the project I have access to: https://harbor-repo.vmware.com/api/v2.0/projects/agamez/repositories

Acceptance criteria

• The Flux OCI support works with robot accounts.

Additional context

N/A

[gfichtenholt]

I am able to use robot accounts with kubeapps flux plug-in in a public OCI repo hosted on demo.goharbor.io without any issues. Can provide server-side log file if needed. A PR with an integration test to that effect is pending https://github.com/vmware-tanzu/kubeapps/pull/5246/files#. I think that supports Antonio's theory. Namely, there IS a way to configure things so that the use case works. If there is a harbor server out there where this fails, then it is more likely than not that the issue with the remote harbor server.

I initially thought that the robot accounts that are given out do not have sufficient privileges for what kubeapps needs whether they be project-specific or not. So I narrowed down the permissions as much as possible to make the use case work. It turned out a single permission was sufficient on the project that contained the OCI repos to make the use case work:
https://github.com/vmware-tanzu/kubeapps/pull/5246/files#diff-415361069319809c0d88b24710344e5efcb8b8572854d83ccb34b7c62abb318c

          {
            "resource": "repository",
            "action": "list"
          }

but then any other single permission by itself (completely unrelated to listing of repositories) also worked such as:

          {
            "resource": "scan",
            "action": "stop"
          }

so I am a bit confused about that right now. Will look into it some more tomorrow.
Regardless, other than figuring out what specific configuration is needed on the remote harbor server, so that we can add it to some doc, I don't see what else I can do for this code-wise. I don't thing there is anything to fix in flux plugin for this.

Some screenshots attached

[gfichtenholt]

please feel free to re-open as you see fit

[gfichtenholt]

Reopening due to:
If I run a test that uses admin account followed by a test that uses a robot account everything runs fine. If I then bounce the kubeapps apis pod and run only the test that uses a robot account that fails. So I think there are a couple of issues:

1. there is some stale in-memory cached state (maybe in ORAS libs) that is set up by one test and somehow affects other tests.
2. There is some likelihood the robot account never worked with kubeapps. To fix it:
   a. We can pursue ppl behind harbor to see if it is possible to use standard API with robot accounts via some server-side config settings OR
   b. might have to introduce a vendor-specific repository-lister