Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export missing records to ES #8337

Closed
saig0 opened this issue Dec 8, 2021 · 4 comments · Fixed by #10697
Closed

Export missing records to ES #8337

saig0 opened this issue Dec 8, 2021 · 4 comments · Fixed by #10697
Assignees
Labels
area/observability Marks an issue as observability related hacktoberfest Marks an issue as a candidate to be a Hacktoberfest contribution kind/feature Categorizes an issue or PR as a feature, i.e. new behavior scope/broker Marks an issue or PR to appear in the broker section of the changelog version:8.2.0-alpha1 Marks an issue as being completely or in parts released in 8.2.0-alpha1 version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0

Comments

@saig0
Copy link
Member

saig0 commented Dec 8, 2021

Is your feature request related to a problem? Please describe.
Currently, the Zeebe Elasticsearch exporter doesn't allow to export all records. The following records are missing:

  • timer
  • message_start_event_subscription
  • process_event
  • deployment_distribution

At the moment, these records are not consumed by a specific application. But they could be useful for debugging a behavior in Zeebe. Sometimes the records can't be read from the log stream because the log is compacted already (default: every 5 minutes).

Describe the solution you'd like
Add the ES indexes for the missing record types.
Extend the exporter configuration to enable/disable the exporting of these records.

Describe alternatives you've considered
No export to ES for these records.

Additional context
This request was raised as an action item of the following incident: https://docs.google.com/document/d/1xQPZbGUG57VOi-u4MI8CZF_I6FbBh0nc_ukSFiZwsNk

@saig0 saig0 added kind/feature Categorizes an issue or PR as a feature, i.e. new behavior scope/broker Marks an issue or PR to appear in the broker section of the changelog Impact: Integration labels Dec 8, 2021
@npepinpe npepinpe added this to Ready in Zeebe Dec 9, 2021
@npepinpe
Copy link
Member

npepinpe commented Dec 9, 2021

I've put it at the back of the ready queue so that it gets done soon-ish, as this can be very helpful when dealing with incidents or debugging other bugs (as described in #8338).

@KerstinHebel KerstinHebel removed this from Ready in Zeebe Mar 23, 2022
@npepinpe npepinpe added area/observability Marks an issue as observability related and removed Impact: Integration labels Apr 11, 2022
@remcowesterhoud remcowesterhoud added the hacktoberfest Marks an issue as a candidate to be a Hacktoberfest contribution label Oct 14, 2022
@korthout korthout added the version:8.2.0-alpha1 Marks an issue as being completely or in parts released in 8.2.0-alpha1 label Nov 1, 2022
@sdorokhova
Copy link
Contributor

Hi @saig0 ,
why exactly did we decide to enable these records by default? For debugging purposes?

@saig0
Copy link
Member Author

saig0 commented Nov 22, 2022

@sdorokhova yes, correct. These records may help if something goes wrong.

Do you face any issues?

@sdorokhova
Copy link
Contributor

We had issues on CI. We had an impression that it's performance related as we started getting more indices in ELS than before. But I disabled them in our tests, so should not be a problem any more.

I would say that in SaaS we may also observe more data usage because of these additional indices, but probably it's fine.

@npepinpe npepinpe added the version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0 label Apr 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/observability Marks an issue as observability related hacktoberfest Marks an issue as a candidate to be a Hacktoberfest contribution kind/feature Categorizes an issue or PR as a feature, i.e. new behavior scope/broker Marks an issue or PR to appear in the broker section of the changelog version:8.2.0-alpha1 Marks an issue as being completely or in parts released in 8.2.0-alpha1 version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants