Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Opensearch exporter fails with 400 on AWS Opensearch service #18251

Closed
megglos opened this issue May 6, 2024 · 0 comments · Fixed by #18253
Closed

Opensearch exporter fails with 400 on AWS Opensearch service #18251

megglos opened this issue May 6, 2024 · 0 comments · Fixed by #18253
Assignees
@megglos
Labels
component/zeebe Related to the Zeebe component/team kind/bug Categorizes an issue or PR as a bug version:8.4.7 Marks an issue as being completely or in parts released in 8.4.7 version:8.5.1 Marks an issue as being completely or in parts released in 8.5.1 version:8.6.0-alpha1 Label that represents issues released on verions 8.6.0-alpha1 version:8.6.0 Label that represents issues released on version 8.6.0

Comments

@megglos
Copy link
Contributor

megglos commented May 6, 2024

Describe the bug

When connecting the zeebe opensearch exporter with the AWS Opensearch service some requests (like removing all ISM policies from zeebe indices) fail with a HTTP 400 without further indication why in the response body.

To Reproduce

  1. create an opensearch service on aws
  2. connect the zeebe opensearch exporter to it

Investigation
Debugging the failing calls by executing them manually via curl revealed the cause is a missing "/" at the beginning of the path like here.

See the following curl example that tries the same request without and with the leading "/" using the --request-target param.

➜  camunda-platform git:(stable/8.4) ✗ curl --location --request-target "_plugins/_ism/remove/zeebe-record*" --request POST 'https://search-operate-XXX.eu-north-1.es.amazonaws.com'  -i \
--header 'Authorization: Basic ****'
HTTP/2 400
server: awselb/2.0
date: Mon, 06 May 2024 06:20:01 GMT
content-type: text/html
content-length: 122

<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>
➜  camunda-platform git:(stable/8.4) ✗ curl --location --request-target "/_plugins/_ism/remove/zeebe-record*" --request POST 'https://search-operate-XXX.eu-north-1.es.amazonaws.com'  -i \
--header 'Authorization: Basic ***'
HTTP/2 200
date: Mon, 06 May 2024 06:20:07 GMT
content-type: application/json; charset=UTF-8
content-length: 981
access-control-allow-origin: *

{"updated_indices":0,"failures":true,"failed_indices":[{"index_name":"zeebe-record_command-distribution_8.5.0_2024-05-03","index_uuid":"uUYcPuUOSES-xEwQNPKr0g","reason":"This index does not have a policy to remove"},{"index_name":"zeebe-record_job_8.5.0_2024-05-03","index_uuid":"BwP_sMgoQAKqzVLPX1mg_w","reason":"This index does not have a policy to remove"},{"index_name":"zeebe-record_process-instance_8.5.0_2024-05-03","index_uuid":"evL-SBEmS3OKmK5RqeaiYA","reason":"This index does not have a policy to remove"},{"index_name":"zeebe-record_variable_8.5.0_2024-05-03","index_uuid":"UBoLC6xMQL6xq5nv8cvwQg","reason":"This index does not have a policy to remove"},{"index_name":"zeebe-record_process_8.5.0_2024-05-03","index_uuid":"m9WaH1M0S86-ZgCx5FVEvw","reason":"This index does not have a policy to remove"},{"index_name":"zeebe-record_process-instance-creation_8.5.0_2024-05-03","index_uuid":"nvjZBDn8TOCQkKeGoJ1XmQ","reason":"This index does not have a policy to remove"}]}%

Expected behavior

Requests should succees like on a bare opensearch instance as used during integration testing.

Log/Stacktrace

Full Stacktrace 

024-05-03 10:45:12 2024-05-03 07:45:12.069 [Broker-0] [zb-fs-workers-0] [Exporter-4] WARN 
2024-05-03 10:45:12       io.camunda.zeebe.broker.exporter.opensearch - Error on exporting record with key 2251799813685249
2024-05-03 10:45:12 io.camunda.zeebe.exporter.opensearch.OpensearchExporterException: Failed to add policy to indices
2024-05-03 10:45:12     at io.camunda.zeebe.exporter.opensearch.OpensearchClient.bulkRemoveISMPolicyToAllZeebeIndices(OpensearchClient.java:275) ~[zeebe-opensearch-exporter-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.exporter.opensearch.OpensearchExporter.updateRetentionPolicyForExistingIndices(OpensearchExporter.java:345) ~[zeebe-opensearch-exporter-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.exporter.opensearch.OpensearchExporter.export(OpensearchExporter.java:96) ~[zeebe-opensearch-exporter-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.lambda$export$5(ExporterContainer.java:188) ~[zeebe-broker-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.util.jar.ThreadContextUtil.runCheckedWithClassLoader(ThreadContextUtil.java:58) ~[zeebe-util-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.util.jar.ThreadContextUtil.runWithClassLoader(ThreadContextUtil.java:34) ~[zeebe-util-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.export(ExporterContainer.java:187) ~[zeebe-broker-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.exportRecord(ExporterContainer.java:174) ~[zeebe-broker-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.broker.exporter.stream.ExporterDirector$RecordExporter.export(ExporterDirector.java:532) ~[zeebe-broker-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.scheduler.retry.BackOffRetryStrategy.run(BackOffRetryStrategy.java:51) ~[zeebe-scheduler-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:94) [zeebe-scheduler-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:47) [zeebe-scheduler-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:122) [zeebe-scheduler-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:130) [zeebe-scheduler-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:108) [zeebe-scheduler-8.4.6.jar:8.4.6]
2024-05-03 10:45:12     at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:227) [zeebe-scheduler-8.4.6.jar:8.4.6]
2024-05-03 10:45:12 Caused by: org.opensearch.client.ResponseException: method [POST], host [https://search-operate-opensearch-2-11-3nkp5dzczb7ntz7gekco3f34em.eu-north-1.es.amazonaws.com/], URI [_plugins/_ism/remove/zeebe-record*], status line [HTTP/1.1 400 Bad Request]
2024-05-03 10:45:12 <html>
2024-05-03 10:45:12 <head><title>400 Bad Request</title></head>
2024-05-03 10:45:12 <body>
2024-05-03 10:45:12 <center><h1>400 Bad Request</h1></center>
2024-05-03 10:45:12 </body>
2024-05-03 10:45:12 </html>

Environment:

  • OS:
  • Zeebe Version: 8.4.x & 8.5.x
  • Configuration:
ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_AUTHENTICATION_PASSWORD=***
ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_AUTHENTICATION_USERNAME=***
ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_BULK_SIZE=1
ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_INDEX_NUMBEROFREPLICAS=2
ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_INDEX_NUMBEROFSHARDS=1
ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_URL=https://search-XXX.eu-north-1.es.amazonaws.com/
ZEEBE_BROKER_EXPORTERS_OPENSEARCH_CLASSNAME=io.camunda.zeebe.exporter.opensearch.OpensearchExporter
@megglos megglos added kind/bug Categorizes an issue or PR as a bug component/zeebe Related to the Zeebe component/team labels May 6, 2024
@megglos megglos self-assigned this May 6, 2024
This was referenced May 6, 2024
Open
Merged
github-merge-queue bot pushed a commit that referenced this issue May 6, 2024
@megglos
fix: add leading slash to opensearch endpoints (#18253)
c9b4a7c 
## Description

While the bare opensearch service allows a missing leading slash, the
AWS service or it's ingress doesn't, returning a 400.

As I don't see an easy way to cover this in our ITs we may consider
following up with a nightly IT on prepared AWS Opensearch instance see
#18252 .

## Related issues

closes #18251
This was referenced May 6, 2024
Merged
Merged
Merged
github-merge-queue bot pushed a commit that referenced this issue May 6, 2024
@camundait
[Backport stable/8.4] fix: add leading slash to opensearch endpoints (#…
ed1313c 
…18274)

# Description
Backport of #18253 to `stable/8.4`.

relates to #18252 #18251
original author: @megglos
github-merge-queue bot pushed a commit that referenced this issue May 6, 2024
@camundait
[Backport stable/8.5] fix: add leading slash to opensearch endpoints (#…
fa65441 
…18275)

# Description
Backport of #18253 to `stable/8.5`.

relates to #18252 #18251
original author: @megglos
megglos added a commit that referenced this issue May 6, 2024
@megglos
[Backport release-8.6.0-alpha1] fix: add leading slash to opensearch …
e1e64a5 
…endpoints (#18276)

# Description
Backport of #18253 to `release-8.6.0-alpha1`.

relates to #18252 #18251
original author: @megglos
@Zelldon Zelldon added version:8.4.7 Marks an issue as being completely or in parts released in 8.4.7 version:8.5.1 Marks an issue as being completely or in parts released in 8.5.1 version:8.6.0-alpha1 labels May 7, 2024
@tmetzke tmetzke added the version:8.6.0 Label that represents issues released on version 8.6.0 label Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@megglos megglos

Labels
component/zeebe Related to the Zeebe component/team kind/bug Categorizes an issue or PR as a bug version:8.4.7 Marks an issue as being completely or in parts released in 8.4.7 version:8.5.1 Marks an issue as being completely or in parts released in 8.5.1 version:8.6.0-alpha1 Label that represents issues released on verions 8.6.0-alpha1 version:8.6.0 Label that represents issues released on version 8.6.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: add leading slash to opensearch endpoints camunda/camunda
3 participants
@megglos @tmetzke @Zelldon