Provide self-managed package-registry that backs integrations UI

[ruflin]

Not all Elastic Stack setups have access to the internet. For the Ingest Manager to fully function, access to https://epr.elastic.co is required. To work around this, it is possible to modify the URL to which Kibana connects. This can be done through xpack.fleet.registryUrl: 'http://localhost:8080'. Today this mainly used for the development of the package registry and testing of new packages before they are released.

IMPORTANT: This config should not be used by users yet and is not supported!

With the above config, it is is possible to run a registry on prem. The package registry with packages is available as docker container under docker.elastic.co/package-registry/distribution:production for production. This is currently compatible with 7.10.0 deployment of Kibana. To run this registry locally, the following command can be used: docker run -p 8080:8080 docker.elastic.co/package-registry/distribution:production

Even though all of the above is possible today, it is not supported and for development only. This issue is to discuss further on what the open issues are to make it supported and how to solve them.

Issues that are still open:

• As long as epr.elastic.co is used, we can guarantee the registry is always the newest version. As soon as it is run on prem, Kibana must enforce a minimal version of the registry to properly work. What is our compatibility matrix?
• What is the recommended upgrade scenario for on prem registry users? How often will these users need / get updates for the packages?
• What version do we recommend on prem users to use?
• What is our recommended way of running the registry?

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[mostlyjason]

Related to elastic/kibana#74614

[ruflin]

@DartShinigami Lets move this to discuss: https://discuss.elastic.co/ Feel free to ping me there. Please also provide the kibana logs for debbuging.

[mostlyjason]

@akshay-saraswat transferring ownership to the integrations team

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[simitt]

When running elastic-agent in airgapped environments, we need to be able to recommend running their own package registry to users. This can be a requirement already today for ECE with apm deployments >= 7.13.
@masci can the work on this be prioritized?

[mtojek]

Currently we have this guide: https://github.com/elastic/package-storage#air-gapped-environments

[simitt]

Thanks @mtojek! Is there an overview around missing steps to make this GA and officially supported?

[mtojek]

Let me pass this question to @akshay-saraswat . I remember that there were few items in the "brainstorm" doc (e.g. proper Docker tags).

[ruflin]

The 4 points I made in the initial issue description still apply.

[akshay-saraswat]

Thanks @mtojek! Is there an overview around missing steps to make this GA and officially supported?

@simitt Please refer this document for the GA plan. With the 7.15 release, we plan to announce the beta version.

[jlind23]

@akshay-saraswat can we think about a GA for EPR in air gapped environment in order to "unlock" APM issues?

[lexlee327]

Is this feature only for ECE and ECK? We have some CSP like tencent & ali. there env can't access to repository.elastic.co, can we find a solution to help them?

…

On Tue, Sep 21, 2021 at 9:24 AM Akshay Saraswat ***@***.***> wrote: Thanks @mtojek <https://github.com/mtojek>! Is there an overview around missing steps to make this GA and officially supported? @simitt <https://github.com/simitt> Please refer this document <https://docs.google.com/document/d/1FvI13YLBpPLG6MtBZvYE2Il0jC2oIygu38nKtpn5JXY> for the GA plan. With the 7.15 release, we plan to announce the beta version. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1178 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANK72C6CSCOY3SQU4UFRZFTUC7NFDANCNFSM47EWTHUQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Solution Architect, China, Mobile : +86 18190768637 Email : ***@***.***

[jlind23]

@lexlee327 running the EPR docker image should be the same as it can be considered as "air gapped" environment if they can not reach repository.elastic.co. @jsoriano @mtojek correct me if I am wrong.

[jsoriano]

I think that there are two different issues here:

• Issues in air-gapped ECE/ECK.
• General availability of a general solution for self-managed package-registry.

For air-gapped ECE/ECK, I understand that the requirement is not the package-registry itself, but to be able to bundle some packages with Kibana so some features work at all without connection to the production registry. There can be other solutions for this as directly bundling the packages in Kibana, or installing them without a registry (elastic/kibana#70582). But if we chose to use the package-registry, it can be already used: it is already GA, as well as most of the required packages.

I am afraid that we are mixing both things together, and this has the risk on one side of unnecessarily delaying the solution of the issues for air-gapped ECE/ECK, and on the other side to put a rush on a general solution for self-managed package registries while there are still some questions on the air.

[simitt]

Packaging some packages with Kibana would solve the issue that the apm and fleet-server cannot even be set up without a connection to the package registry. The apm use case should be fine with this solution. But when users want to create custom agent policies to enroll their remote agents via the centralized fleet server wouldn't that also be a problem when no other integrations are available?

[m-adams]

Bundling in the pure basics as a fallback seems like a good idea to stop immediate errors for basic usecases during the getting started experience offline.
EPR is experimental/beta at the moment but IMO we haven't solved this until

• it's GA
• it can be run by ECE and ECK (we have a growing list of things we are asking the user to host outside our orchestration platforms)
• Any other gotchas that assume internet, endpoint artifacts? Fleet agent upgrade? Etc. Maybe it's all solved by EPR now

[jsoriano]

Updating docs for 7.16 in elastic/observability-docs#1287.

[akshay-saraswat]

I believe, with self-managed EPR GA in 7.16, we have accomplished the scope of this ticket. Does anyone have any objection if I close this issue?