Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow disabling an exporter via an API #18030

Closed
8 tasks done
Tracked by #16126
deepthidevaki opened this issue Apr 29, 2024 · 1 comment · Fixed by #18807
Closed
8 tasks done
Tracked by #16126

Allow disabling an exporter via an API #18030

deepthidevaki opened this issue Apr 29, 2024 · 1 comment · Fixed by #18807
Assignees
Labels
component/zeebe Related to the Zeebe component/team kind/feature Categorizes an issue or PR as a feature, i.e. new behavior

Comments

@deepthidevaki
Copy link
Contributor

deepthidevaki commented Apr 29, 2024

For dual-region, we have an ElasticSearch exporter for each region separately. During failover, we want to disable the exporter to the failed region to ensure that Zeebe can continue exporting to the surviving exporter and compact the logs.

As discussed in the poc, we will built this feature on top of the cluster topology which allows to keep track of the latest state of the exporter and make changes in a systematic and observable manner.

The state of exporter is persisted in the cluster

  1. deepthidevaki
  2. kind/task scope/broker
    deepthidevaki
  3. version:8.6.0-alpha2
    deepthidevaki

I can disable an exporter by its id with out restarting the cluster

  1. component/zeebe kind/task scope/broker version:8.6.0-alpha2
    deepthidevaki
  2. component/zeebe kind/task scope/broker version:8.6.0-alpha2
    deepthidevaki
  3. component/zeebe kind/task scope/broker version:8.6.0-alpha2
    deepthidevaki
  4. component/zeebe kind/task scope/broker version:8.6.0-alpha2
    deepthidevaki
  5. component/zeebe kind/task scope/broker
    deepthidevaki
@deepthidevaki deepthidevaki self-assigned this Apr 29, 2024
@deepthidevaki deepthidevaki added the kind/feature Categorizes an issue or PR as a feature, i.e. new behavior label Apr 29, 2024
@deepthidevaki deepthidevaki added the component/zeebe Related to the Zeebe component/team label May 8, 2024
@deepthidevaki
Copy link
Contributor Author

Follow up discussion of #18512 (review) with @npepinpe

We decided to go with the following semantics. The feasibility of a technical solution needs to be evaluated.

  1. The exporter state in the dynamic config is enabled. But the broker restarts with exporter removed from the static configuration.

In this case, the removed exporter will be automatically set to disabled by Zeebe. The state of the exporter will be removed from the runtime (which is also the current behavior.) When the exporter is re-added, it has to be explicitly enabled via the API.

Deleting the exporter completely from the dynamic config should be evaluated later. Currently due to the need for keeping metadataVersion for correctness for next re-enabling, it is required to keep the exporter in the dynamic config.

  1. The exporter doesn't exist in the dynamic config. But a broker restarts with a new exporter added to the static configuration.

The newly added exporter will be enabled by default.

Technical solution idea: On restart, brokers compare the dynamic config and the static config and updates its own state in the dynamic config accordingly. Each brokers updates its own state and gossips this change. The merge logic ensures that all brokers eventually has the same dynamic configuration state. We have to ensure that this wouldn't conflict with any concurrent scaling or config change operations that updates the dynamic config.

github-merge-queue bot pushed a commit that referenced this issue May 29, 2024
## Description

* Refactor open-api spec for cluster management api to extract re-usable
schema that are useful for exporter api and other configuration
management that will be built on top of the same infrastructure.
* Add `ExportersEndpoint` for handling request to disable exporter

As discussed in #18421, we started with a simple API. No additional
query is added. For now cluster status query can be used to monitor the
status of the request and the status of exporters.

## Related issues

closes #18421
closes #18030
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/zeebe Related to the Zeebe component/team kind/feature Categorizes an issue or PR as a feature, i.e. new behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant