Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Umbrella Issue for improved and Utility EventSources provided for V2 #729

Closed
csviri opened this issue Dec 1, 2021 · 3 comments · Fixed by #720
Closed

Umbrella Issue for improved and Utility EventSources provided for V2 #729

csviri opened this issue Dec 1, 2021 · 3 comments · Fixed by #720
Assignees
Projects
Milestone

Comments

@csviri
Copy link
Collaborator

csviri commented Dec 1, 2021

Event Sources provides functionality to get notifications, thus trigger reconciliation if a dependent resource changes. Withing this issue the topic is revisited, and will try to improve and provide implementation of event sources that would help to cover most of the use cases to a certain level of abstraction.

The goal is to have "batteries included", so users could use and/or easily implement their own custom event sources utilizing the ones we provide. This is intended to be the first design / iteration, plan is to evolve these based on user feedback.

Event Source Categories

We can divide event sources into two big categories:

  1. Event Sources for Kuberentes Resource or InformerEventSource
    This event source covers the watching and caching the Kubernetes resources. Already present in the system. Provides a
    cache of the observed resources too.
  2. Non Kubernetes (external) Resources. Think of a object that is not managed by Kubernetes it self, from Database users, to git repositories, etc. But still here we communicate with an API. Ideally still through well defined REST API, thus managing resources in a declarative way. Based how these objects and related events are handled we can divide this to two sub-categories.
    1. Pull based - when we have to basically poll the resource to get the actual state, in regular intervals.
    2. Push based - when a notification in some form is received about the changes of the resource. Like a webhook or sqs queue, etc.

Polling Event Source for Pull Based Updates

For this purpose two implementations are provided:

  1. PerResourcePollingEventSource - Periodically calls a provided function for each custom resource.
  2. PollingEventSource - Is for use cases when resources can be queried from an api, but it is done for a generic way. Like listing some object with realted tags. Then these are mapped to ResourceID-s.

Both of these provide caches of the resources. The events are propagated only if resources changed (checked by equals method on pojos).

Inbound Event Sources for Push Based Updates

Inbound event sources are for the case when we receive some notifications about the resources. For now we don't support specific systems, but provide basic implementation that could be used or extended to easily support any notification system. Like webhooks, or messaging systems.

  1. SimpleInboundEventSource - tyically for use cases when the notification is not resource based. This event source make it just easy to propagate any event into the system. Does not provide caching.

  2. CachingInboundEventSource - Typically for use cases when the notification is resource based, in other works we receive the whole resource within the event. This implementation provides caching.

Note that these event sources also provide addtional functionalities, like filtering of events.

Abstract Event Sources to Support Custom Ones

CachingFilteringEventSource - is an abstract event source (serves as a base class for some above), that supports caching and event filtering.

Custom Resource Event Notifications for Event Source

There are cases when an Event Source needs to be aware of the custom resources managed by the system, for example:

  • TimerEventSource - wants to cancel all the timed tasks if a custom resource is deleted
  • PerResourcePollingEventSource - also needs to stop polling for the custom resource if it's not present in the system. But also want't start polling in case a custom resource appeared (or reached a certain state - see predicate in the implementation).

For this purpose the ResourceEventAware was introduced. If an event sources implements this interface will receive events about the custom resources in the system. Allowing to implement logic based on that.

See also:

#651
#665
#666
#673

@csviri
Copy link
Collaborator Author

csviri commented Dec 2, 2021

@metacosm @lburgazzoli @adam-sandor @heesuk-ahn if you have time pls take a look, and let me know what do you think.

@heesuk-ahn
Copy link
Contributor

heesuk-ahn commented Dec 3, 2021

This seems like a good first step. 👍

It is good that the user can directly create a Custom EventSource through the abstraction layer, or that Cache or Filter can be directly injected from the user through the constructor.

This point is considered to be an advantage in that it is possible to access Cache or Filter outside of the framework during development.

It is not yet clear how each EventSource will be used, but I'll take a closer look at this part. :)

@csviri
Copy link
Collaborator Author

csviri commented Dec 3, 2021

Next step will include these to examples.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Project
  
Done
Development

Successfully merging a pull request may close this issue.

2 participants