Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix][io] Fix es index creation #22654

Merged
merged 2 commits into from
May 6, 2024
Merged

[fix][io] Fix es index creation #22654

merged 2 commits into from
May 6, 2024

Conversation

nodece
Copy link
Member

@nodece nodece commented May 6, 2024

Motivation

#14383 introduces the event-time-based index name, it works fine.

This behavior has been broken by #14805.

Elasticsearch sink yaml:

configs:
    elasticSearchUrl: "http://localhost:9200"
    indexName: "test-index-%{+yyyy-MM-dd}"
    createIndexIfNeeded: true
    compatibilityMode: ELASTICSEARCH

When the index name equals to the test-index-%{+yyyy-MM-dd}, the es index creation fails:

co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/index] failed: [invalid_index_name_exception] Invalid index name [test-index-%{+yyyy-MM-dd}], must be lowercase
	at co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse(ElasticsearchTransportBase.java:345) ~[elasticsearch-java-8.12.1.jar:?]
	at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:147) ~[elasticsearch-java-8.12.1.jar:?]
	at co.elastic.clients.elasticsearch.ElasticsearchClient.index(ElasticsearchClient.java:1141) ~[elasticsearch-java-8.12.1.jar:?]
	at org.apache.pulsar.io.elasticsearch.client.elastic.ElasticSearchJavaRestClient.indexDocument(ElasticSearchJavaRestClient.java:163) ~[gurFS0VR_Lh-tuSoWDLguA/:?]
	at org.apache.pulsar.io.elasticsearch.ElasticSearchClient.indexDocument(ElasticSearchClient.java:190) [gurFS0VR_Lh-tuSoWDLguA/:?]
	at org.apache.pulsar.io.elasticsearch.ElasticSearchSink.write(ElasticSearchSink.java:139) [gurFS0VR_Lh-tuSoWDLguA/:?]
	at org.apache.pulsar.functions.instance.JavaInstanceRunnable.sendOutputMessage(JavaInstanceRunnable.java:469) [pulsar-functions-instance.jar:3.3.0-SNAPSHOT]
	at org.apache.pulsar.functions.instance.JavaInstanceRunnable.handleResult(JavaInstanceRunnable.java:431) [pulsar-functions-instance.jar:3.3.0-SNAPSHOT]
	at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:352) [pulsar-functions-instance.jar:3.3.0-SNAPSHOT]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

The expected index name should be equal to test-1-2024-05-06(eventTime: System.currentTimeMillis()).

Modifications

  • Use the index passed in instead of reading the index from config.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@nodece
[fix][io] Fix es index creation
22ad97e 
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label May 6, 2024
dao-jun
dao-jun approved these changes May 6, 2024
View reviewed changes
Copy link
Member

@dao-jun dao-jun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@shibd shibd added type/bug The PR fixed a bug or issue reported a bug release/3.2.3 release/3.0.5 ready-to-test labels May 6, 2024
@shibd shibd closed this May 6, 2024
@shibd shibd reopened this May 6, 2024
@lhotari lhotari requested a review from nicoloboschi May 6, 2024 08:36
@nodece
Fix test
c72506e 
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
@nodece nodece merged commit efcedf6 into apache:master May 6, 2024
48 of 50 checks passed
@nodece nodece self-assigned this May 6, 2024
@nodece nodece added this to the 3.3.0 milestone May 6, 2024
Technoboy- pushed a commit that referenced this pull request May 8, 2024
@nodece @Technoboy-
[fix][io] Fix es index creation (#22654)
b86ebaa 
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
nodece added a commit to nodece/pulsar that referenced this pull request May 13, 2024
@nodece
[fix][io] Fix es index creation (apache#22654)
06946d3 
Signed-off-by: Zixuan Liu <nodeces@gmail.com>

(cherry picked from commit efcedf6)
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
nodece added a commit to nodece/pulsar that referenced this pull request May 13, 2024
@nodece
[fix][io] Fix es index creation (apache#22654)
fccaacc 
Signed-off-by: Zixuan Liu <nodeces@gmail.com>

(cherry picked from commit efcedf6)
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
nodece added a commit that referenced this pull request May 13, 2024
@nodece
[fix][io] Fix es index creation (#22654) (#22701)
f1daa75 
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
@nodece
Copy link
Member Author

nodece commented May 13, 2024

Opened a PR to cherry-pick this to the branch-3.0: #22701

nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request May 15, 2024
@nodece @nikhil-ctds
[fix][io] Fix es index creation (apache#22654) (apache#22701)
6d4f606 
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
(cherry picked from commit f1daa75)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request May 16, 2024
@nodece @srinath-ctds
[fix][io] Fix es index creation (apache#22654) (apache#22701)
d6500ee 
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
(cherry picked from commit f1daa75)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RobertIndie RobertIndie RobertIndie approved these changes

@dao-jun dao-jun dao-jun approved these changes

@shibd shibd shibd approved these changes

@nicoloboschi nicoloboschi Awaiting requested review from nicoloboschi

Assignees

@nodece nodece

Labels
cherry-picked/branch-3.0 cherry-picked/branch-3.2 doc-not-needed Your PR changes do not impact docs ready-to-test release/3.2.3 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
3.3.0
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@nodece @RobertIndie @dao-jun @shibd @lhotari @Technoboy-