Sync with Elasticsearch main branch

[mashhurs]

Description

This PR introduces changes aligned with Elasticsearch main branch which a big change, JDK 21 requirement.

Author's checklist

• Rebase after landing Apply Elasticsearch GeoIP changes and sync with Elasticsearch 8.16. #170
• Cut 8.x branch which will be compatible with 8.x stack versions
• Run and check the build CI: https://buildkite.com/elastic/logstash-filter-elastic-integration-build/builds/392
  • validates the separated branches

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b3334ad

History

• 💔 Build #365 failed c11bbb9
• 💔 Build #359 failed b53b26c
• 💔 Build #356 failed 9078935

[jsvd]

LGTM

[jsvd]

should we move gradle.properties ES TREEISH to main?

[mashhurs]

should we move gradle.properties ES TREEISH to main?

Not necessarily from functional point of view as we are able to enforce the ES treeish with env var. If we need it for a visibility (make sure we don't make any mistake while developing), I am thinking to update the GH PR template which should guide 8.x and future release and probably I will also add a note to a plugin publisher.
How do you think?

[jsvd]

The issue I see is that, for a PR against the main branch of this plugin, for example, the main branch of ES is not being tested:

Resolved ELASTICSEARCH_TREEISH: 8.x
....
#31 34.81 > Task :downloadElasticsearchSourceZip
#31 34.81 Download https://github.com/elastic/elasticsearch/archive/refs/heads/8.x.zip

So we may need to force TREEISH to main in this branch, either through CI or the gradle.properties

[mashhurs]

The issue I see is that, for a PR against the main branch of this plugin, for example, the main branch of ES is not being tested:

Resolved ELASTICSEARCH_TREEISH: 8.x
....
#31 34.81 > Task :downloadElasticsearchSourceZip
#31 34.81 Download https://github.com/elastic/elasticsearch/archive/refs/heads/8.x.zip

So we may need to force TREEISH to main in this branch, either through CI or the gradle.properties

I have a PR to separate branches and give better names. How do you think if I add there? I have validated on my local to cover your concern if it is about failure.
Commit: d89a233

• Unit test job

  - label: ":hammer: Unit tests with LS 8.x-SNAPSHOT & ES main :docker:"
    command:
      - .buildkite/scripts/run_tests.sh
    env:
      ELASTIC_STACK_VERSION: "8.x"
      INTEGRATION: false
      SNAPSHOT: true
      ELASTICSEARCH_TREEISH: main

Local result:

logstash-1  | Finished in 4.37 seconds (files took 3.5 seconds to load)
logstash-1  | 74 examples, 0 failures
logstash-1  | 
logstash-1  | Randomized with seed 56776
logstash-1  | 
logstash-1 exited with code 0

• Integration test job

  - label: ":hammer: Integration tests with LS 8.x-SNAPSHOT & ES main :docker:"
    command:
      - .buildkite/scripts/run_tests.sh
    env:
      ELASTIC_STACK_VERSION: "8.x"
      INTEGRATION: true
      LOG_LEVEL: "info"
      SECURE_INTEGRATION: true
      SNAPSHOT: true
      ELASTICSEARCH_TREEISH: main

Local result:

logstash-1       |       raises an exception when condition is met
logstash-1       | 
logstash-1       | Finished in 14.59 seconds (files took 3.35 seconds to load)
logstash-1       | 49 examples, 0 failures
logstash-1       | 
logstash-1       | Randomized with seed 61530
logstash-1       | 
logstash-1 exited with code 0

[jsvd]

I'm fine with using #177 to improve the CI. LGTM