Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a lightning-fast global strategy #2058

Merged
merged 8 commits into from
Mar 3, 2021

Conversation

nicolaferraro
Copy link
Member

@nicolaferraro nicolaferraro commented Feb 19, 2021

Fix #1799

This is the best strategy I could think about, and it comes with no copies or redundant information.

Basically, the behavior of Camel K is now the following when running an integration with a global operator in another namespace:

  • If there's an integrationplatform in the local integration namespace, then Camel K behaves as today
  • If there's no integrationplatform in the local integration namespace, then the operator looks for an integrationplatform in the operator namespace (or it creates one when possible, i.e. OpenShift currently) and all the hard work (like checking kits and building images) is done in the global operator namespace

In the new context, when I run an integration, tipically I have:

  • Integration in the user namespace
  • IntegrationKit in the operator namespace
  • Build in the operator namespace
  • CamelCatalog in the operator namespace
  • IntegrationPlatform in the operator namespace

I.e. the disconnection happens at integration level, all other pieces remain local (in the operator namespace).
To locate the kit and all other platform resources, a new status -> platformNamespace has been added to the Integration and it represents where all supporting resources are located.

On OpenShift, the pull-secret trait also adds the system:image-puller role (thanks @astefanutti) to the integration service account, so no image transfer between namespaces: all (cluster-wide) integrations with the same set of dependencies will point to the same imagestream.

This allows to have, by default, a single builder with warmed up cache, that also builds images for the whole cluster, so images are shared and time to run the integrations is drastically reduced.

Missing parts:

  • E2E tests
  • Handle explicit references to local kits

Release Note

A cluster-wide global Camel K operator now has the same (or better) performance than local Camel K operators

@nicolaferraro nicolaferraro added kind/feature New feature or request status/wip Work in progress labels Feb 19, 2021
@lburgazzoli
Copy link
Contributor

How does this strategy handle the case where you programmatic create an IntegrationKit ?

@astefanutti
Copy link
Member

Woot! For future references, next steps towards the perfect build system is #592, and possibly #1784.

@nicolaferraro
Copy link
Member Author

How does this strategy handle the case where you programmatic create an IntegrationKit ?

Well, no problems if the integrationKit is co-located with the integration platform (i.e. it's in the operator namespace).

But probaly we should allow also local references... But at the same time we need not to complicate the model too much... thinking...

@lburgazzoli
Copy link
Contributor

How does this strategy handle the case where you programmatic create an IntegrationKit ?

Well, no problems if the integrationKit is co-located with the integration platform (i.e. it's in the operator namespace).

Yep but normal user should not create stuffs in the operator namespace

But probaly we should allow also local references... But at the same time we need not to complicate the model too much... thinking...

Maybe instead of having

status:
    platformNamespace

Each linked resource should report the information about the namespace ?

status:
    kit:
      # optional, current if omitted
      namespace: 
      name:

I guess it won't be too hard :D to have cascading lookup.

@nicolaferraro nicolaferraro force-pushed the global-strategy-2 branch 5 times, most recently from 94fe1a3 to c88586d Compare March 2, 2021 08:03
@nicolaferraro
Copy link
Member Author

I've deprecated "kit" and added an "integrationKit" object reference property in both spec and status, and implemented the suggested behavior.

The only caveat is that a kit should always follow a platform (same namespace) in order to be built, with the exception of "external" kits that can also be created in the same namespace as the integration (e2e tests included).

@nicolaferraro nicolaferraro removed the status/wip Work in progress label Mar 2, 2021
@nicolaferraro nicolaferraro merged commit 631f67d into apache:master Mar 3, 2021
@nicolaferraro nicolaferraro mentioned this pull request Apr 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implementing a global build strategy
4 participants