Skip to content

Add configurable indexing event decoder#1220

Draft
jwils wants to merge 1 commit into
joshuaw/json-ingestion-test-migrationfrom
joshuaw/indexer-ingestion-adapters
Draft

Add configurable indexing event decoder#1220
jwils wants to merge 1 commit into
joshuaw/json-ingestion-test-migrationfrom
joshuaw/indexer-ingestion-adapters

Conversation

@jwils
Copy link
Copy Markdown
Collaborator

@jwils jwils commented May 30, 2026

Why

The indexer lambda currently assumes SQS payload bodies are JSON Lines. To support ingestion sources that serialize events differently, the payload decoding step needs to be configurable before the normal ElasticGraph event validation and indexing pipeline runs.

What

  • Add an indexer.indexing_event_decoder extension setting using the existing runtime extension loader pattern.
  • Add a default ElasticGraph::Indexer::IndexingEventDecoder::JSONLines decoder to preserve current behavior.
  • Wire elasticgraph-indexer_lambda SQS processing through the configured decoder while keeping warehouse lambda compatibility.
  • Add RBS, focused specs, README docs, and regenerated config schema.

Verification

  • script/type_check
  • bundle exec rspec elasticgraph-indexer/spec/unit/elastic_graph/indexer/config_spec.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer_spec.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/sqs_processor_spec.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/lambda_function_spec.rb elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda/lambda_function_spec.rb
  • bundle exec standardrb elasticgraph-indexer/lib/elastic_graph/indexer.rb elasticgraph-indexer/lib/elastic_graph/indexer/config.rb elasticgraph-indexer/lib/elastic_graph/indexer/event_id.rb elasticgraph-indexer/lib/elastic_graph/indexer/indexing_event_decoder.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer/config_spec.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer_spec.rb elasticgraph-indexer/spec/support/example_extensions/indexing_event_decoder.rb elasticgraph-indexer_lambda/lib/elastic_graph/indexer_lambda/lambda_function.rb elasticgraph-indexer_lambda/lib/elastic_graph/indexer_lambda/sqs_processor.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/sqs_processor_spec.rb
  • script/update_config_artifacts --verify

Stack

Current PR is marked with ->.

@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch 2 times, most recently from f31c540 to dbc7b07 Compare May 31, 2026 05:24
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 31, 2026

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants