[exporter/elasticsearchexporter]: Add dynamic index option for both logs and traces #16094
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
Hello! I'm in need of this feature too, do we have a forecast of when the merge will take place? |
|
@MatheusGeiger that is great to hear! I am still waiting for the code review. |
JaredTan95
reviewed
Dec 9, 2022
Foresight Summary
View More Details✅ tracegen workflow has finished in 59 seconds (2 minutes 18 seconds less than
|
| Job | Failed Steps | Tests | |
|---|---|---|---|
| publish-latest | - 🔗 | N/A | See Details |
| build-dev | - 🔗 | N/A | See Details |
| publish-stable | - 🔗 | N/A | See Details |
⭕ build-and-test-windows workflow has finished in 9 seconds (30 minutes 25 seconds less than main branch avg.) and finished at 13th Apr, 2023.
| Job | Failed Steps | Tests | |
|---|---|---|---|
| windows-unittest-matrix | - 🔗 | N/A | See Details |
| windows-unittest | - 🔗 | N/A | See Details |
✅ telemetrygen workflow has finished in 1 minute 14 seconds and finished at 13th Apr, 2023.
| Job | Failed Steps | Tests | |
|---|---|---|---|
| build-dev | - 🔗 | N/A | See Details |
| publish-latest | - 🔗 | N/A | See Details |
| publish-stable | - 🔗 | N/A | See Details |
✅ check-links workflow has finished in 1 minute 55 seconds (⚠️ 59 seconds more than main branch avg.) and finished at 13th Apr, 2023.
| Job | Failed Steps | Tests | |
|---|---|---|---|
| changed files | - 🔗 | N/A | See Details |
| check-links | - 🔗 | N/A | See Details |
✅ changelog workflow has finished in 2 minutes 5 seconds and finished at 13th Apr, 2023.
| Job | Failed Steps | Tests | |
|---|---|---|---|
| changelog | - 🔗 | N/A | See Details |
✅ prometheus-compliance-tests workflow has finished in 3 minutes 13 seconds (3 minutes 7 seconds less than main branch avg.) and finished at 13th Apr, 2023.
| Job | Failed Steps | Tests | |
|---|---|---|---|
| prometheus-compliance-tests | - 🔗 | N/A | See Details |
✅ load-tests workflow has finished in 6 minutes 36 seconds (3 minutes 54 seconds less than main branch avg.) and finished at 13th Apr, 2023.
| Job | Failed Steps | Tests | |
|---|---|---|---|
| setup-environment | - 🔗 | N/A | See Details |
| loadtest (TestIdleMode) | - 🔗 | N/A | See Details |
| loadtest (TestBallastMemory|TestLog10kDPS) | - 🔗 | N/A | See Details |
| loadtest (TestMetric10kDPS|TestMetricsFromFile) | - 🔗 | N/A | See Details |
| loadtest (TestMetricResourceProcessor|TestTrace10kSPS) | - 🔗 | N/A | See Details |
| loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs) | - 🔗 | N/A | See Details |
| loadtest (TestTraceBallast1kSPSWithAttrs|TestTraceBallast1kSPSAddAttrs) | - 🔗 | N/A | See Details |
| loadtest (TestTraceAttributesProcessor) | - 🔗 | N/A | See Details |
✅ e2e-tests workflow has finished in 13 minutes 27 seconds and finished at 13th Apr, 2023.
| Job | Failed Steps | Tests | |
|---|---|---|---|
| kubernetes-test (v1.25.3) | - 🔗 | N/A | See Details |
| kubernetes-test (v1.24.7) | - 🔗 | N/A | See Details |
| kubernetes-test (v1.26.0) | - 🔗 | N/A | See Details |
| kubernetes-test (v1.23.13) | - 🔗 | N/A | See Details |
✅ build-and-test workflow has finished in 36 minutes 28 seconds (10 minutes 8 seconds less than main branch avg.) and finished at 13th Apr, 2023.
| Job | Failed Steps | Tests | |
|---|---|---|---|
| govulncheck | - 🔗 | N/A | See Details |
| setup-environment | - 🔗 | N/A | See Details |
| build-examples | - 🔗 | N/A | See Details |
| check-collector-module-version | - 🔗 | N/A | See Details |
| checks | - 🔗 | N/A | See Details |
| check-codeowners | - 🔗 | N/A | See Details |
| correctness-metrics | - 🔗 | N/A | See Details |
| correctness-traces | - 🔗 | N/A | See Details |
| integration-tests | - 🔗 | N/A | See Details |
| lint-matrix (receiver-0) | - 🔗 | N/A | See Details |
| lint-matrix (receiver-1) | - 🔗 | N/A | See Details |
| lint-matrix (processor) | - 🔗 | N/A | See Details |
| lint-matrix (exporter) | - 🔗 | N/A | See Details |
| lint-matrix (extension) | - 🔗 | N/A | See Details |
| lint-matrix (connector) | - 🔗 | N/A | See Details |
| lint-matrix (internal) | - 🔗 | N/A | See Details |
| lint-matrix (other) | - 🔗 | N/A | See Details |
| unittest-matrix (1.20, receiver-0) | - 🔗 | N/A | See Details |
| unittest-matrix (1.20, receiver-1) | - 🔗 | N/A | See Details |
| unittest-matrix (1.20, processor) | - 🔗 | N/A | See Details |
| unittest-matrix (1.20, exporter) | - 🔗 | N/A | See Details |
| unittest-matrix (1.20, extension) | - 🔗 | N/A | See Details |
| unittest-matrix (1.20, connector) | - 🔗 | N/A | See Details |
| unittest-matrix (1.20, internal) | - 🔗 | N/A | See Details |
| unittest-matrix (1.20, other) | - 🔗 | N/A | See Details |
| unittest-matrix (1.19, receiver-0) | - 🔗 | N/A | See Details |
| unittest-matrix (1.19, receiver-1) | - 🔗 | N/A | See Details |
| unittest-matrix (1.19, processor) | - 🔗 | N/A | See Details |
| unittest-matrix (1.19, exporter) | - 🔗 | N/A | See Details |
| unittest-matrix (1.19, extension) | - 🔗 | N/A | See Details |
| unittest-matrix (1.19, connector) | - 🔗 | N/A | See Details |
| unittest-matrix (1.19, internal) | - 🔗 | N/A | See Details |
| unittest-matrix (1.19, other) | - 🔗 | N/A | See Details |
| lint | - 🔗 | N/A | See Details |
| unittest (1.20) | - 🔗 | N/A | See Details |
| unittest (1.19) | - 🔗 | N/A | See Details |
| cross-compile (darwin, amd64) | - 🔗 | N/A | See Details |
| cross-compile (darwin, arm64) | - 🔗 | N/A | See Details |
| cross-compile (linux, 386) | - 🔗 | N/A | See Details |
| cross-compile (linux, amd64) | - 🔗 | N/A | See Details |
| cross-compile (linux, arm) | - 🔗 | N/A | See Details |
| cross-compile (linux, arm64) | - 🔗 | N/A | See Details |
| cross-compile (linux, ppc64le) | - 🔗 | N/A | See Details |
| cross-compile (windows, 386) | - 🔗 | N/A | See Details |
| cross-compile (windows, amd64) | - 🔗 | N/A | See Details |
| build-package (deb) | - 🔗 | N/A | See Details |
| build-package (rpm) | - 🔗 | N/A | See Details |
| windows-msi | - 🔗 | N/A | See Details |
| publish-check | - 🔗 | N/A | See Details |
| publish-stable | - 🔗 | N/A | See Details |
| publish-dev | - 🔗 | N/A | See Details |
| rotate-milestone | - 🔗 | N/A | See Details |
*You can configure Foresight comments in your organization settings page.
|
@dmitryax this pr needs one more approval. :) thank you in advance |
|
just to prevent this pr being staled. |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
dmitryax
reviewed
Mar 8, 2023
dmitryax
approved these changes
Mar 21, 2023
… potential null pointer issue resolved
|
is there anything else, which needs to be done here? :) |
dmitryax
reviewed
Mar 28, 2023
|
Hello there, It's a important feature blocking for us to deploy opentelemetry with a production load without having a huge single index. I volonteer to help if something more is needed 😄 |
Tbh I don't think there is things to do as it is already tested. I am also waiting for approver to merge it as well. |
|
@vsellier I will solve any merge conflict and commit it. |
|
Hi, |
dmitryax
approved these changes
May 16, 2023
|
Merged. Sorry for the delay |
varunraiko
pushed a commit
to varunraiko/opentelemetry-collector-contrib
that referenced
this pull request
May 17, 2023
…ogs and traces (open-telemetry#16094) The elasticsearchexporter of current version takes each one static index for both logs and traces. The capability of dynamic indexing based on attributes (elasticsearch.index.prefix, elasticsearch.index.suffix) will help users to delegate the final index. Until now, what I have seen solar is the usage of ingestion pipeline dedicatedly for rewriting index, which is the only way to write data to multiple indexes. This way, users have multiple options to route their data to a prefixed or suffixed index by using resource processor or attribute processor in sidecar collector/ main collector or even by adding it as static attribute in application instrumentation.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
The elasticsearchexporter of current version takes each one static index for both logs and traces. The capability of dynamic indexing based on attributes (
logs_index_prefix,logs_index_suffix,traces_index_prefix,traces_index_suffix,elasticsearch.index.prefix,elasticsearch.index.suffix) will help users to delegate the final index.Until now, what I have seen solar is the usage of ingestion pipeline dedicatedly for rewriting index, which is the only way to write data to multiple indexes.
This way, users have multiple options to route their data to a prefixed or suffixed index by using resource processor or attribute processor in sidecar collector/ main collector or even by adding it as static attribute in application instrumentation.
Link to tracking Issue:
#5854
Testing:
Test with test-purposed elastic search exporter with attributes defined above and check if the incoming bulk request has arrived in mock server with the expected prefixed&suffixed index.
Documentation:
Added in Readme