ES|QL PoC.

[mashhurs]

Do not review or merge, it is just PoC which helps during the discussion.

Logs

╭─logstash ~ 
╰─➤  bin/logstash -f config/input-elasticsearch.conf --enable-local-plugin-development
Using system java: /.sdkman/candidates/java/current/bin/java
Sending Logstash logs to /logstash/logs which is now configured via log4j2.properties
[2025-03-25T15:50:29,683][INFO ][logstash.runner          ] Log4j configuration path used is: /logstash/config/log4j2.properties
[2025-03-25T15:50:29,686][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2025-03-25T15:50:29,686][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"9.1.0", "jruby.version"=>"jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.5+11-LTS on 21.0.5+11-LTS +indy +jit [arm64-darwin]"}
[2025-03-25T15:50:29,687][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2025-03-25T15:50:29,710][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000` (logstash default)
[2025-03-25T15:50:29,710][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000` (logstash default)
[2025-03-25T15:50:29,710][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-nesting-depth` configured to `1000` (logstash default)
[2025-03-25T15:50:29,723][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because command line options are specified
[2025-03-25T15:50:29,983][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2025-03-25T15:50:30,132][INFO ][org.reflections.Reflections] Reflections took 50 ms to scan 1 urls, producing 149 keys and 522 values
[2025-03-25T15:50:31,128][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2025-03-25T15:50:31,140][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>10, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1250, "pipeline.sources"=>["/logstash/config/input-elasticsearch.conf"], :thread=>"#<Thread:0x6b2bf074 /logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"}
[2025-03-25T15:50:31,387][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.25}
Query mode: esql
[2025-03-25T15:50:36,523][INFO ][logstash.inputs.elasticsearch][main] `search_api => auto` resolved to `search_after` {:elasticsearch=>"9.0.0-SNAPSHOT"}
[2025-03-25T15:50:36,523][INFO ][logstash.inputs.elasticsearch][main] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2025-03-25T15:50:36,523][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2025-03-25T15:50:36,525][INFO ][logstash.inputs.elasticsearch.esql][main][218332e6b55e7bb722a1fda7d144b153ba72aa4712d8e163fd0f659c6f001f4d] ES|QL executor starting
[2025-03-25T15:50:36,528][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Response: #<Elasticsearch::API::Response:0x396f8d6a @response=#<Elastic::Transport::Transport::Response:0x3b74b4ba @headers={"took-nanos"=>"13314125", "x-elasticsearch-async-is-running"=>"?0", "x-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "transfer-encoding"=>"chunked"}, @body={"took"=>13, "is_partial"=>false, "columns"=>[{"name"=>"log.file.device_id", "type"=>"keyword"}, {"name"=>"log.file.fingerprint", "type"=>"keyword"}, {"name"=>"log.file.inode", "type"=>"keyword"}, {"name"=>"log.file.path", "type"=>"keyword"}, {"name"=>"log.file.path.text", "type"=>"text"}, {"name"=>"log.level", "type"=>"keyword"}, {"name"=>"log.logger", "type"=>"keyword"}, {"name"=>"log.offset", "type"=>"long"}, {"name"=>"log.origin.file.line", "type"=>"long"}, {"name"=>"log.origin.file.name", "type"=>"keyword"}, {"name"=>"log.origin.file.name.text", "type"=>"text"}, {"name"=>"log.origin.function", "type"=>"keyword"}, {"name"=>"log.source", "type"=>"keyword"}], "values"=>[["16777233", "f028282db1d14a80a6c16214c1127db142bce486bc1a8ebc5a2415f24c8c0436", "179534999", "/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson", "/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson", "info", "publisher_pipeline_output", 163549, 146, "pipeline/client_worker.go", "pipeline/client_worker.go", "github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run", "system/metrics-default"]]}, @status=200>>
columns class: Array
values class: Array
columns: [{"name"=>"log.file.device_id", "type"=>"keyword"}, {"name"=>"log.file.fingerprint", "type"=>"keyword"}, {"name"=>"log.file.inode", "type"=>"keyword"}, {"name"=>"log.file.path", "type"=>"keyword"}, {"name"=>"log.file.path.text", "type"=>"text"}, {"name"=>"log.level", "type"=>"keyword"}, {"name"=>"log.logger", "type"=>"keyword"}, {"name"=>"log.offset", "type"=>"long"}, {"name"=>"log.origin.file.line", "type"=>"long"}, {"name"=>"log.origin.file.name", "type"=>"keyword"}, {"name"=>"log.origin.file.name.text", "type"=>"text"}, {"name"=>"log.origin.function", "type"=>"keyword"}, {"name"=>"log.source", "type"=>"keyword"}]
values: ["16777233", "f028282db1d14a80a6c16214c1127db142bce486bc1a8ebc5a2415f24c8c0436", "179534999", "/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson", "/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson", "info", "publisher_pipeline_output", 163549, 146, "pipeline/client_worker.go", "pipeline/client_worker.go", "github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run", "system/metrics-default"]
values: {"log.file.device_id"=>"16777233", "log.file.fingerprint"=>"f028282db1d14a80a6c16214c1127db142bce486bc1a8ebc5a2415f24c8c0436", "log.file.inode"=>"179534999", "log.file.path"=>"/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson", "log.file.path.text"=>"/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson", "log.level"=>"info", "log.logger"=>"publisher_pipeline_output", "log.offset"=>163549, "log.origin.file.line"=>146, "log.origin.file.name"=>"pipeline/client_worker.go", "log.origin.file.name.text"=>"pipeline/client_worker.go", "log.origin.function"=>"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run", "log.source"=>"system/metrics-default"}
Mapped Data: {"log.file.device_id"=>"16777233", "log.file.fingerprint"=>"f028282db1d14a80a6c16214c1127db142bce486bc1a8ebc5a2415f24c8c0436", "log.file.inode"=>"179534999", "log.file.path"=>"/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson", "log.file.path.text"=>"/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson", "log.level"=>"info", "log.logger"=>"publisher_pipeline_output", "log.offset"=>163549, "log.origin.file.line"=>146, "log.origin.file.name"=>"pipeline/client_worker.go", "log.origin.file.name.text"=>"pipeline/client_worker.go", "log.origin.function"=>"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run", "log.source"=>"system/metrics-default"}
mapped_entry class: Hash
mapped_entry value: {"log.file.device_id"=>"16777233", "log.file.fingerprint"=>"f028282db1d14a80a6c16214c1127db142bce486bc1a8ebc5a2415f24c8c0436", "log.file.inode"=>"179534999", "log.file.path"=>"/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson", "log.file.path.text"=>"/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson", "log.level"=>"info", "log.logger"=>"publisher_pipeline_output", "log.offset"=>163549, "log.origin.file.line"=>146, "log.origin.file.name"=>"pipeline/client_worker.go", "log.origin.file.name.text"=>"pipeline/client_worker.go", "log.origin.function"=>"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run", "log.source"=>"system/metrics-default"}
{
                    "log.level" => "info",
           "log.file.device_id" => "16777233",
         "log.file.fingerprint" => "f028282db1d14a80a6c16214c1127db142bce486bc1a8ebc5a2415f24c8c0436",
          "log.origin.function" => "github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run",
                   "@timestamp" => 2025-03-25T22:50:36.557093Z,
                   "log.offset" => 163549,
    "log.origin.file.name.text" => "pipeline/client_worker.go",
                   "log.source" => "system/metrics-default",
         "log.origin.file.line" => 146,
                "log.file.path" => "/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson",
               "log.file.inode" => "179534999",
                   "log.logger" => "publisher_pipeline_output",
         "log.origin.file.name" => "pipeline/client_worker.go",
                     "@version" => "1",
           "log.file.path.text" => "/elastic-agent-9.0.0-beta1-darwin-aarch64/data/elastic-agent-aa8178/logs/elastic-agent-20250306-1.ndjson"
}