[Epic] Tagging Community/Partner Integrations

[jamiehynds]

Intro

Elastic currently supports 300+ integrations via Elastic Agent, to ingest data from common data sources across security and observability use cases. All integrations are treated equally from a support standpoint regardless of where the integration came from. We have three main sources of integrations:

• Elastic integration teams (including contractors)
• Vendors who build integrations for their products under our partnership program
• Community users who develop and contribute integrations

While we really appreciate and value community contributions, we are limited in our ability to test these integrations as we typically don't have access to the data source and don't have a technology partnerships in place with the vendor. Similarly, for vendor/partner developed integrations, while we review the PR, we didn't build the integration and can't guarantee the same quality as Elastic developed integrations.

Proposal

In order to set expectations with users on the quality and support they can expect from an integration, we should highlight the source of the integration within integration documentation and on the integrations UI. We will continue to provide best effort support and don't want situations where users adopt community developed integrations that are completely unsupported. However, expectations should be set that Elastic has not developed these integrations, does not have access to the data source and to expect possible parsing/mapping/performance issues.

We need to finalise wording and ensure we set user expectations correctly, but something similar to Hashicorp's community supported wording is what we're aiming for:

**Community Supported** – the S3 storage backend is supported by the community. While it has undergone review by HashiCorp employees, they may not be as knowledgeable about the technology. If you encounter problems with them, you may be referred to the original author.

To Consider

• All integrations, regardless of the source, reside in the elastic/integrations repo. Should we consider moving community and/or technology partner integrations to a separate repo? How would Elastic Package Registry handle this?
• Beyond a section within the documentation, could we add attributes to the packages and make those attributes visible in Fleet. e.g. Developed by
• Within Fleet could we add a filter based on integration source? e.g. Elastic, Vendor, Partner (unsure on the value of this, but worth considering at least).

Tasks

• Add new field to package-spec to indicate owner/developer type
  Add owner.type field to indicate package support level package-spec#568
  Set owner.type field as required package-spec#604
• Update SEI packages to use new field
  sei: Update packages to format_version 3.0.0 #7810
  Change owner type for SEI packages not owned by Elastic #8035
• Document new field
  [Fleet] Display owner.type field from package manifest on integration overview page kibana#167981
  https://github.com/elastic/integration-docs/issues/255

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[jamiehynds]

@jsoriano - FYI as this will likely require changes on the elastic-package side. @andrewkroh has some thoughts.

@SubhrataK I'm hoping we could extend this across all integrations across security and observability. Would be great to get your input.

[jsoriano]

• All integrations, regardless of the source, reside in the elastic/integrations repo. Should we consider moving community and/or technology partner integrations to a separate repo? How would Elastic Package Registry handle this?

Not all integrations reside on the integrations repo, there are some integrations like APM or endpoint that reside on other repositories.

Package publication in the package registry works by triggering a job in internal-ci. This job could be triggered from any CI pipeline that supports triggering Jenkins jobs. With this approach we could only support jobs owned by Elastic, as some credentials are required to trigger jobs in internal-ci.

With this approach we could allow contributors to open PRs to the current integrations repository, or we could have another repository with community contributors. In both cases the repository would need to be owned by Elastic so it has access to internal-ci, and we should approve the merges that would trigger package publication.

Something else to take into account is that all packages published now are signed with Elastic keys. This is the main reason to use internal-ci for publication now.

A more advanced approach would imply to implement a new public API to publish packages, that would replace the job in internal-ci. This would allow external contributors to directly publish their own packages, signed with their own keys.

If we start publishing packages from different sources it will be also interesting to add build information in the packages to know where the source comes from elastic/package-spec#446.

FYI as this will likely require changes on the elastic-package side.

In principle no changes would be required on elastic-package as it doesn't publish packages. Unless we decide to implement the public API and give support to it in elastic-package.

Beyond a section within the documentation, could we add attributes to the packages and make those attributes visible in Fleet. e.g. Developed by

We currently have the owner section in the manifest. It currently supports github teams, what might be enough. Elastic packages would be owned by elastic teams, while community/partner packages wouldn't be.

Within Fleet could we add a filter based on integration source? e.g. Elastic, Vendor, Partner (unsure on the value of this, but worth considering at least).

Yes, I think we could work on the UI for this. We could maybe add a badge to differentiate Elastic from Community contributions. cc @juliaElastic

[ebeahan]

As a first step, could we implement some sort of tagging in package-spec? Tagging integrations as community would set expectations about testing and supportability and include the community supported expectations in the integration's docs.

Yes, I think we could work on the UI for this. We could maybe add a badge to differentiate Elastic from Community contributions. cc @juliaElastic

Same tagging could visually differentiate in the UI.

[SubhrataK]

We are working on a similar project under Observability.

Key points that I would like to highlight, which are being discussed as part of this project are:

1. We will not own and support integrations that are contributed by the community/partner. This shall be made explicit to the community/partner while signing the legal agreement. And to our customers through documentation and also through tags or description in the integrations catalog UI.
2. We should have a tag to identify and filter the community/partner developed integrations in the catalog for the users to filter.

CC: @lalit-satapathy and @rameshelastic

[taylor-swanson]

Hi all, here are my current thoughts/ideas/comments:

Regarding the separate community repo:

• Discussions on this are out of scope for this particular issue, but I'll give my thoughts anyway. In general, I'm for this idea. It separates community integrations entirely from offical Elastic ones, which helps keep the noise down in the main integrations repo.

With that said, I'm in favor of having an explicit flag for the type of owner for the package. I quickly drafted a new field in the package spec for this:

owner:
  type: object
  additionalProperties: false
  properties:
    type:
      description: Describes who owns the package and the level of support that is provided.
      type: string
      enum:
        - "elastic"
        - "partner"
        - "community"
    github:
      ...

(If anyone has better suggestions for naming or the description, I'm open to ideas)

This would be a required field moving forward.

Alternatively, as @jsoriano mentioned, we could derive this "owner type" from the github owner field already present:

• Begins with elastic/: Elastic-owned integration
• Anything else: Community-owned integration

That leaves partner integrations. We could dedicate a group within elastic/ for partner integrations (elastic/partner-integrations). When matched against that group, then it becomes a Partner-owned integration. However, That contradicts this statement:

We will not own and support integrations that are contributed by the community/partner

A partner may develop an integration entirely separate from us, so using an Elastic-owned github group probably won't not work here. If we did want to call out partner integrations differently than community integrations, we would have to come up with a method to do that.

As far as the Kibana UI work is concerned, I'll have to defer to someone who understands this better. To me, having a dedicated field that can be filtered on seems more reliable than trying to derive the owner type from the github owner. I could be wrong, though.

One other issue I thought of with this approach is keeping the logic in sync between different systems which need to know the owner type. For example, docs generation will need to know the owner type in order to inject a statement about the support level of the package. Likewise, the Kibana UI will need to know the owner type in order to filter packages. These mechanisms exist in entirely different repos across different teams. I'm sure there are other places, too, and I have concerns about keeping the logic in sync, especially if there are a lot of edge cases that come up with partners vs community packages.

[jamiehynds]

@jsoriano would you mind reviewing Taylor's write up above and let us know your thoughts? As we ramp up on community and partner developed integrations, we're aiming to get this capability done for 8.10. Thanks!

[taylor-swanson]

I've created a PR of the proposed change to package-spec here: elastic/package-spec#568

[jsoriano]

Hey, sorry for the late reply.

I am fine with the owner.type field as a temporary solution as part of the "poor man's marketplace". It will work while we (as Elastic) review all packages that are published, and we remember to check this field. There are some things to define as what to assume by default when the field is not set. I don't think we can enforce this field as required because that would be a breaking change, and wouldn't work for packages already published. We can discuss about these details in the PR.

In the long run this should probably be part of the publishing pipeline. Ideally we would have a public API to publish packages, and the authentication used there would help us to distinguish the type of owner. We could also use the signing keys. We are currently the publishers of all packages, and all packages are signed with our keys, in the future external users could use their own keys.

A middle ground here that we could implement sooner, based on the owner handle, would be to have a list of partners somewhere, then elastic/* owners would indicate an Elastic integration, owners in the list of partners would indicate partners integrations, and the rest would be community.

ccing @mrodm in case you have other ideas or preferences.

[taylor-swanson]

Status update 2023-10-03:

• The owner.type field has been added as a required field to package-spec in the top-level manifest.yml file in a package
• All SEI packages have been updated to include an owner type
• Packages from other teams are being updated to include owner type as they move to package-spec 3.0.0

Next steps:

• Expose the owner type field in the package details in the Kibana Fleet UI (similar to how other fields, such as subscription level and license are presented)
• Determine if we need dedicated documentation or knowledge base article on owner types (describes support levels, expectations of developer support, etc)
• Add filters to Kibana Fleet UI to filter on owner type (future work, likely not under this issue)

[nimarezainia]

@kpollich @juliaElastic please see @taylor-swanson comment here.

[kpollich]

Expose the owner type field in the package details in the Kibana Fleet UI (similar to how other fields, such as subscription level and license are presented)

This should be relatively low effort. I created elastic/kibana#167981 to track this.

Add filters to Kibana Fleet UI to filter on owner type (future work, likely not under this issue)

This will probably be more involved, and we may want this filtering to be done at the API level in EPR. Agree on punting this to a follow-up where we can define the behavior better.

[jamiehynds]

Thanks @kpollich .

Determine if we need dedicated documentation or knowledge base article on owner types (describes support levels, expectations of developer support, etc)

This is important as we'd like users to get more information on the different types. Do you recommend we host this within our integration docs, and we can link to those from the Fleet UI? Alternatively, we could do something similar to the changelog, where we expose a pop-up with explainations on the owner types?

[kpollich]

Do you recommend we host this within our integration docs, and we can link to those from the Fleet UI?

This seems like a good idea to me. That way if we ever add new owner types we can update the docs and we won't have churn to update a modal in Kibana. I'll add a note to that issue above saying we should link to a docs page once one exists.

[taylor-swanson]

@jamiehynds, with @kpollich (and team) working on implementing changes in the Kibana UI, I think the last remaining work would be to update our documentation.

I could see us making a change to elastic-package to generate a callout/section in the readme to note who developed the package and expected support levels. In order to do that, though, I would need some verbiage around each of the support levels. Would we want legal or perhaps the docs team to write something up? Would that be something you could track down?

I think once it's in the README, that information will be propagated throughout the ecosystem (i.e. it should show up on docs.elastic.co, the fleet integration page in Kibana, etc).

[taylor-swanson]

@kpollich, I think for elastic/kibana#167981, the intent was to display it as data point on the right hand side along with other package info. While I still think this is a good idea, I think having a block in the main text of the package description would also be very import to have. I was thinking something on the lines of what is displayed for pre-release packages. I'll let UX decide how it wants decorate the block depending on the package owner/support level type. I originally thought this was rendered in the README markdown somehow, but I was wrong. We'll have to something similar to docs.elastic.co/integrations, as well.

I pointed to the pre-release tag, but I'm not as concerned about that. Again, I'll defer to UX if they have preferences either way. Mainly, we want a clear and upfront description of who developed this package and to set expectations of support levels.

[jamiehynds]

I'd worry about overloading the integration description with various labels, banners, etc. I'd be inclined to include it as a data point under the Details section. We'll also need to link to a page within our docs (or in-product) to explain what each 'Developed By' category means - maybe an info or question icon beside Developed By. I'm working on the wording of those descriptions.

We may need to put some thought into integrations developed by partners. Do we just say Developed By: Partner or include the partner/vendor name, and support contact details (which requires additional fields).

[taylor-swanson]

To be clear, here is what I was thinking about the banner (this would be for the docs site, but would apply to the Kibana UI as well):

Disclaimer: Wording is NOT final.

My concern with the sidebar is it will be overlooked/missed by users, but I can agree that overloading the description with too many labels, banners, etc can quickly become a problem.

[taylor-swanson]

We may need to put some thought into integrations developed by partners. Do we just say Developed By: Partner or include the partner/vendor name, and support contact details (which requires additional fields).

Good point. As far as what's in the manifest, the only information we'd have access to currently is owner.github, which in most cases points back to SEI. I'd imagine we'd have to add a separate name field under owner.

[taylor-swanson]

Status update: The Kibana task has been completed and will be in 8.12. The remaining work is being tracked by the Docs team (see issue linked in the description).

[ebeahan]

EPR added support for owner.type in v1.23.0, and this version is deployed on EPR prod:

$ curl -sL https://epr.elastic.co/
{
  "service.name": "package-registry",
  "service.version": "1.23.0"
}

But the owner.type property is still not appearing for packages:

$ curl -sL https://epr.elastic.co/package/1password/1.26.0/ | jq '.owner'
{
  "github": "elastic/security-external-integrations"
}

Investigation still ongoing into why EPR isn't providing owner.typer (elastic/package-registry#1126).

Kibana handles the missing value gracefully in 8.12 and the Developed By field doesn't appear in if EPR doesn't provider owner.type. Once the value is supplied by EPR, Developed By should begin appearing with no other changes.

[111andre111]

Just a coincidental question on that. Once it would appear in Kibana, would it be shown as well in docs.elastic.co?
#6569 (comment)

[taylor-swanson]

@111andre111, the docs team is looking into that now, but as of writing, I don't believe they have added anything in yet that displays this information. Discussions on how the information should be displayed and the resulting work is tracked here: elastic/integration-docs#255

[alaudazzi]

@taylor-swanson @111andre111 @jamiehynds @SubhrataK
FYI the Obs doc team is taking this a priority for January. We will be creating a dedicated page with the level of support and reference it from the integration pages.
Relates to https://github.com/elastic/integration-docs/issues/255

[taylor-swanson]

Thank you for the update, @alaudazzi! This is great news.

[ebeahan]

elastic/package-registry#1126 is now resolved.

Examples of the different owner type values as displayed in EPM:

[taylor-swanson]

The owner type/support levels are now being presented on docs.elastic.co (see https://github.com/elastic/integration-docs/issues/255#issuecomment-1895744461), which wraps up the remaining work for this epic.