Discussion: What is the correct way to reference a community plugin via spec

[erezrokah]

With #632 I expected the following spec to work for a plugin that is hosted on the repo https://github.com/cloudquery/cq-source-plaid

kind: source
spec:
  name: plaid
  version: v1.0.0
  destinations: [postgresql]
  path: cloudquery/cq-source-plaid
  spec:
    client_id: ${PLAID_CLIENT_ID}
    secret: ${PLAID_SECRET}
    access_token: ${PLAID_ACCESS_TOKEN}

However it doesn't and only the following works:

kind: source
spec:
  name: plaid
  version: v1.0.0
  destinations: [postgresql]
  path: cloudquery/plaid
  spec:
    client_id: ${PLAID_CLIENT_ID}
    secret: ${PLAID_SECRET}
    access_token: ${PLAID_ACCESS_TOKEN}

This is not an issue with said PR, but just how the current code expected to work, see tests:

plugin-sdk/clients/source/v0/source_test.go

Line 23 in 384b195

Path: "yevgenypats/test",

I find it a bit unintuitive that the path doesn't match the repo.
It this expected or we should make path: cloudquery/cq-source-plaid work instead?

Maybe someone wants to put their plugin under <org>/my-cq-plugin-repo-test-5? With the current approach we don't support it. Also it's harder for users to understand where is the plugin hosted

[hermanschaaf]

Yeah, this was a decision a few months ago when we moved to v1. It is documented here. I agree, seeing it in use now, it would be better if configs could be explicit.

I think the simplest backwards-compatible change here would be to allow both formats, i.e. cloudquery/cq-source-plaid and cloudquery/plaid should both work. (The latter would expand to cloudquery/cq-source-plaid) The cq-source-* repo naming requirement would then still be in effect, but configs can be more explicit about it.

The other option is to cut our losses and release a breaking change that requires the full repo path, dropping the cq-source-* and cq-destination-* naming requirements. Maybe we can do that in the future, but I don't see a big need for it right now.

[erezrokah]

Got it and thank you for pointing me to the docs. I think that if we want to enforce a repository naming convention we should still be able to do it with cloudquery/cq-source-plaid by verifying the format of the repo name.
That way the config is explicit and we have consistent naming.

I'll let @yevgenypats comment before we make any further decisions here

[yevgenypats]

I would allow only one format and this should be solved by documentation eventually anyway so I don't think it's a big hurdle. Also, better only to support one way as otherwise it can cause verious issues with retries of different URLs and so on. There is no big need to specify cq-source as this was the whole point because we already specify those as kind: source/destination

I don't see any real issue with the current way and there would be other issues with the otherway around so we should just document it, maybe add it to the FAQ and potentially this can be also solved when we will have much more community plugins and better documentation website where you can just copy/paste the path

[erezrokah]

Let's close this for now and if we see that others get confused like me we can re-visit