Skip to content

Stateless for Self-Managed: Easier to operate, reduced indexing costs #104

@dbrimley

Description

@dbrimley

Description:

This outcome solves the critical problem of high TCO and inflexible scaling for large, self-managed deployments. Building upon segment-based replication, it allows Self-Managed customers to use low-cost object storage (e.g., S3, GCS, Azure Blob) as the system of record and scale indexing and search compute resources independently. This dramatically reduces indexing costs and provides the operational flexibility needed to manage petabyte-scale environments efficiently.

Key Capabilities:

  • Enable configuration of certified S3-compatible object store as the primary repository for data, where users wish to use non-CSP object stores and keep data on-prem.
  • Enable the use of the 3 major CSP Object Stores.
  • Provide distinct node roles or tiers to allow for the independent scaling of indexing compute and search compute.
  • Develop UI (in Kibana) and APIs for configuring and managing object store connections and compute tiers.
  • Provide clear migration and upgrade paths for existing Self-Managed Enterprise customers to adopt the stateless architecture.
  • Improved Sharding experience, with new APIs to split & shrink shards in place.

Stateless for Self-Managed will be packaged as an Enterprise feature and be distributed via the usual Elastic Stack packages. ECK will also be capable of running Stateless clusters.

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

Status

In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions