ChRIS plugin apps' logs streaming and replay architecture

[jbernal0019]

Problem:

Currently container logs associated to a ChRIS plugin app running on a remote compute environment (Docker, Podman, Swarm, Kubernetes, SLURM) are obtained by CUBE by continuously polling the compute environment API through pfcon at very small time intervals to achieve near-realtime updates. The container logs are stored and updated directly in the relational Postgres DB as a long text field of a plugin instance table. When a user in the ChRIS UI wants to look at the logs panel for a running plugin instance the UI also continuously polls CUBE at very small time intervals to get the log updates.

This architecture has many obvious flaws. Among them:

• The two involved constant polling steps unnecessary consume network and compute resources which can strain the ChRIS system.
• Doesn't scale well. For instance the Kubernetes API proxy could be overwhelmed by the polling mechanism affecting the compute cluster performance.
• The relational DB is not adequate to store the unstructured logs data. This also hampers the possibility of offline analyses of the plugins logs.

A new architecture should be devised to fix or mitigate the above issues.

Solution requirements:

1. Stream container logs in near-realtime (some reasonable delay is acceptable) when required by a user all the way from the container scheduler (Docker, Podman, Swarm, Kubernetes, SLURM) to the ChRIS UI.
2. Replay events and container logs. Logs should also be stored in an appropriate data store for later replay and perhaps simple offline analyses.
3. Zero events loses due to infrastructure errors.
4. Minimize logs loses due to infrastructure errors.
5. Scale to thousands of containers running in a compute cluster.
6. Prioritize simpler to implement, easier to maintain architectures with minimal human and compute resources footprint. Add minimal amount of new components to the existing system. If possible reuse existing components. Eg. reuse existing RabbitMQ message queue instead of adding an additional complex Kafka message queue just for this.

[jbernal0019]

As a first proposition we should consider the pros and cons of simpler solutions that then could get incrementally more complex as required.
Discuss simpler sound architectures and then increase complexity as required to meet the problem requirements.

An initial proposition coud be:

sequenceDiagram
    participant UI as ChRIS UI
    participant CUBE as SSE (CUBE)
    participant RQ as RabbitMQ
    participant DB as MongoDB
    participant FL as Fluentd
    participant RT as Runtime (Docker / Podman / Swarm / Kubernetes / SLURM)
    participant PF as pfcon

   UI ->> CUBE: /plugins/instances/<id>/logs/
   CUBE -->> UI: Stream logs (SSE)
   CUBE ->> RQ: Celery queue subscription
   RQ ->> CUBE:  near-realtime job's logs updates
   CUBE ->> DB: fetch a job's logs
   DB->> CUBE:  job's logs replay, recovery,<br/>offline analyses
   FL->> RQ: publish jobs' logs from runtimes
    FL->> DB: store jobs' logs from runtimes
    RT -->> FL: runtime container/jobs' logs
     PF -->> RT: schedule plugin apps,<br/>set logs to Fluentd

Loading

Notes:

• Docker engine comes with a fluentd logs driver out of the box. And we can use a demonset for this conteinerized service for Kubernetes.
• On one hand MongoDB should be enough for logs storage, replay and simple offline analyses. On the other hand ElasticSearch is a data store that is commonly used to manage application logs. It enables more complex offline analyses. However it has a higher resource footprint, learning curve and more complex setup for a production cluster.
• MongoDB ≥ 3.6 supports change streams if it’s running as a replica set (even a single-node replica set). This could be used to stream the logs directly from MongoDB to CUBE. But this doesn't scale as well as using a message queue and could lose more messages.
• RabbitMQ message queue can scale very large but we could swap it for an even larger scale Kafka message queue. However RabbitMQ is already part of the current system and has better Celery support than Kafka https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/index.html. Celery support for Kafka is experimental and importantly (for concurrency management)lacks the inspect and manage workers at runtime features.
• The message queue must be secured as the compute cluster could be in a different remote network.

[chhsiao1981]

1. fluentd is written in ruby. kafka is a much better choice.

[chhsiao1981]

2. no need rabbitmq～
   kafka/fluentd is already sufficient for CUBE-SSE to get the data.
   rabbitmq is the non-scalable version of kafka/fluentd.

[chhsiao1981]

3. there is no need of celery for CUBE-SSE. pure confluent-kafka-python is already sufficient for CUBE-SSE.

[chhsiao1981]

a caveat for kafka/fluentd/nats.io is that it is not good to frequently create/delete topics. this causes huge burden on kraft and the corresponding metadata-cluster.

However, the log may last for hours/days and may be huge. pure-mem based redis-stream may not be a good fit either.

a possible solution is the mix of persistent storage + streaming, as streaming is for the newest 1000 lines of data, and we get the full logs from persistent storage.

[chhsiao1981]

given that kafka/ fluentd / nats.io is not good to frequently create/delete topics, perhaps the best method for kafka / fluentd / nats.io is the same as @jbernal0019's figure, as simply aggregating logs to persistent storage. methods to do SSE is TBD. However, the source of the queue for SSE is not from persistent storage, but directly from kafka/fluentd/nats.io.

[chhsiao1981]

as for the queue for SSE, I would recommend redis-stream as pure-mem streaming. with the constraint that we store at most 1000 lines, and expected that each line with around 200 chars, it's expected that each log is at most around 10M mem and redis-stream should be good enough. If we need to scale-out, we can do sharding for the redis-stream, or we can directly use redis-cluster.

The only caveat is that we may need to check whether redis-stream is included in the community version.

[chhsiao1981]

The implementation of having data in redis-stream is pulling directly from kafka / fluentd / nats.io, not from persistent storage. When pulling the data, we can store in n redis-servers for redundancy (n = 2 should be good enough).

[chhsiao1981]

For the record, the following is the previous design:

FNNDSC/ChRIS_ultron_backEnd#663

[jbernal0019]

@rudolphpienaar @jennydaman @Sandip117
Below is my new compute logs and events architecture proposition. A new set of components independent of pfcon. Pfcon would remain the control plane for job submission, job data download and job and data deletion. The nice thing I recently found out is that we can use Redis to fan out logs per job id!

The Event Forwarder async watcher process, Log Consumer / Streaming Worker and Status Consumer Service - worker process need to be implemented as custom long running consumer or producer Kafka worker processes.

flowchart LR

%% =========================
%% SOURCES
%% =========================
subgraph S[Compute Runtime]
    K8S[Kubernetes Jobs / Pods]
    DCKR[Docker Containers]
end

%% =========================
%% EVENT PIPELINE
%% =========================
subgraph E[Event Pipeline]
    EF[Event Forwarder Daemon - async watcher process]
    K1[(Kafka: job-status-events)]
    ST[Status Consumer Service - worker process]
    Q[Task Queue - Celery / RQ]
end

%% =========================
%% LOG PIPELINE
%% =========================
subgraph L[Log Pipeline]
    FB[Fluent Bit / Vector - log agent]
    K2[(Kafka: job-logs)]
    ES[(Elasticsearch / OpenSearch)]
end

%% =========================
%% REAL-TIME STREAMING LAYER
%% =========================
subgraph R[Real-time Log Streaming Layer]
    LC[Log Consumer / Streaming Worker]
    RD[(Redis Pub/Sub)]
    DJ[Django SSE Service - auth + streaming]
    UI[Web UI - log viewer]
end

%% =========================
%% EVENT FLOWS
%% =========================
K8S --> EF
DCKR --> EF
EF --> K1
K1 --> ST
ST --> Q

%% =========================
%% LOG FLOWS
%% =========================
K8S --> FB
DCKR --> FB
FB --> K2
K2 --> ES

%% =========================
%% STREAMING FLOWS
%% =========================
K2 --> LC
LC --> RD
RD --> DJ
DJ --> UI

Loading

[chhsiao1981]

hi @jbernal0019 ～

redis pub/sub is an excellent choice in this context～

However, one caveat for redis pub/sub: cannot retrieve the logs before subscription.

I believe redis pub/sub is necessary in this context, but also requires django/intent-server have an api for UI to retrieve elk to get the logs before subscription.

[jbernal0019]

@chhsiao1981 Yes, we will keep the logs in Kafka for some time for replay. They will also be persisted in ES. Will need to implement the typical distributed system stuff in the custom consumers (retries + idempotency, logs batching, logs compression, event replay, etc...).

[jennydaman]

Premature optimization is admittedly fun, but for the sake of productivity, before choosing the components and designing an architecture, we need to set a scope and identify requirements.

Questions about Requirements

1. Expected peak load? E.g. might there be 100 users simultaneously trying to observe logs in real-time? 1,000? 1,000,000?
2. Do we favor "correctness" which requires that stdout+stderr be typed as byte-streams, or "practicality" and impose a limitation that logs must be valid UTF-8?
3. Do we want a search feature? If so, how good should it be? I.e. search solutions can be: simple substring search, typo tolerant search, vector search, real-time search suggestions, ...
4. How important is it that a full system be "easy" to setup from scratch?
5. Should we leverage an existing log storage backend where available, which optimizes platform integration and resource usage? Note: greater support for integration with existing log storage backends compounds the complexity of setup, making requirement 4 and 5 exponentially difficult to prioritize simultaneously. Discussion: keep CUBE telemetry/logging backend swappable for OpenShift-native options CHRIS_docs#50
6. What is our preference regarding whether to choose well-established solutions (e.g. Kafka, ElasticSearch) v.s. current-gen and next-gen technologies (e.g. OpenTelemetry, Apachy Iggy)?
7. Which platforms do we aim to support? More concretely, could we drop support for Docker and focus entirely on Kubernetes & OpenShift?
8. Jorge has talked about two kinds of data streams: logs and events. Do we want to think about metrics e.g. CPU and memory usage? These metrics should also be streamed through the platform and visible on the UI.

Thought Experiment: KISS

KISS stands for "keep it simple, stupid."

Kafka is not a secret sauce which magically adds scalability and event-drivenness. I can imagine that a system with fewer than 1,000 concurrent streams can work perfectly well, if not better, without Kafka. In more concrete terms, I mean that in a situation where there are fewer than 1,000 human users trying to observe in real-time the logs of plugin instances running at any given time (with anywhere from 1,000–10,000 plugin instances running in the background, logs which are not being actively observed by human end users are less burdensome on the system because there is no [real-time] streaming requirement).

Component Choices

I strongly believe in doing more research upfront to choose the right tool for the job. I also strongly prefer adopting current and next-gen technologies over using well-established solutions, which tend to be much more resource-intensive, steeper learning curve, and legacy patterns in both API design and deployment architecture. Moreover, from a business perspective, these choices can make it easier for us to answer the question: "what makes us different from others?"

[chhsiao1981]

I believe the purpose of the this discussion is to be able to show the logs on the ChRIS-ui in real-time. Therefore:

1. My understanding of OpenTelemetry (collector) is receiving the data, processed the data as TSDB based data, and then push to TSDB-based storage (elasticsearch / prometheus / grafana / etc.) It's more like the "log-pipeline" without "real-time log-streaming layer" in Jorge's figure. The way it achieves real-time updating is more like grafana's approach: routinely retrieve the data every n seconds. (I'm actually ok with Grafana's approach, with the context of TSDB-based data.)
2. I believe the reason why we want see the logs on the ChRIS-ui in real-time is to have the users able to easily understand the progress of the instance. If we just want to debug the instance, we simply can do posthoc display AFTER the instance finished with success/failure, and we don't need real-time updating of the logs on ChRIS-ui.

[chhsiao1981]

My understanding of OpenTelemetry / Grafana / Prometheus is that they are more likely for system-admin～
Therefore, it's ok to sometimes have downtime / lose data.

However, because our scenario is to show the logs on the ChRIS-ui TO THE USERS, we should choose the components with minimum data lost / minimum downtime.

My understanding of Jennings' n-user question is: "whether we want to have a horizontally-scalable system". I would vote for "yes".

I had a very bad experience with logstash, as that we need to plan ahead how to store the data to really have the claimed performance / resilience. My memory shows that their suggestion for changing the data-schema is simply setting-up another one and migrate the data, which is basically not feasible for big-data. If we do want to use elasticsearch / opensearch, then I believe we need to investigate the appropriate storage for elasticsearch / opensearch.

[jennydaman]

The requirement for "real-time" i think is exaggerated, I agree that it is acceptable and sometimes a better architecture to "retrieve the data every n seconds."

[jennydaman]

Your understanding of OpenTelemetry, Grafana, and Prometheus have some grounding in reality but is otherwise a big misunderstanding. Data loss is implicated with Prometheus but not OpenTelemetry.

OpenTelemetry and Grafana are not inherently lossy technologies. Prometheus is somewhat lossy, it collects "metrics" data. It's not only inconsequential but infeasible to have accurate metrics data so it is not something the technology is concerned with.

Grafana is front-end software for visualizing data and has no role in the collection nor storage of data. Grafana is also the name of a
company which offers observability SaaS.

OpenTelemetry is an ecosystem around the collection of observability data which is comprised of logs, metrics, and traces. Metrics are only one fraction of the concerns handled by OpenTelemetry. In contrast to metrics, for both logs and especially traces it is important to have 100% uptime, coverage, and retention. Traces are actually for the benefit of software developers, not just system-admins.

[jennydaman]

Log Storage Options

• ElasticSearch is the well-established solution. OpenSearch is a fork of ElasticSearch. Both have advanced search features, but are resource-intensive, especially on memory requirements.
• Grafana Loki is another well-established solution. It is more focused on being simple and lightweight. It is much easier to operate and interface with than ElasticSearch, and it supports basic substring search. However, it lacks advanced search e.g. semantic search features.
  • LokiStack is an integrated component of Red Hat OpenShift. Discussion: keep CUBE telemetry/logging backend swappable for OpenShift-native options CHRIS_docs#50
• QuickWit is moreso a "next-gen" solution, though with good industry adoption. It is highly performance-optimized, easy to deploy, horizontally scalable, OpenTelemetry (OTel)-native. QuickWit is written in Rust and built upon the tantivy engine, all developed by the same team.
• OpenObserve is another "next-gen" solution, also built upon the tantivy engine. Like QuickWit, it is written in Rust, stores logs, otel-native, performance-optimized, easy to deploy, and horizontally scalable. Additionally, OpenObserve does compression of storage, handles metrics and traces, real-user monitoring, and supports SQL + PromQL.

Decision Tree for Log Storage

• I want to use industry-standard technologies, I have zero risk tolerance.
  • I have 100s of GB of memory at my disposal. Compute resources are cheap and plentiful. --> ElasticSearch or OpenSearch
  • I want the software to be simple and efficient. --> Loki
• I want to explore more interesting options which have supposedly been engineered to fix the pain points of industry-standard options.
  • High performance text search is my only need. --> QuickWit
  • I want to minimize storage costs and/or am interested in metrics storage. --> OpenObserve

[jbernal0019]

@rudolphpienaar @jennydaman @Sandip117 @chhsiao1981

Considering the idea that (at the moment at least) we don't need to scale really large for the realtime logs and events as there won't be many concurrent users watching them in realtime from the UI I've implemented a new POC that uses simpler Readis Streams instead of Kafka. This is keeping in mind that there will still be many jobs running at any time (thousands) though. This significantly reduces setup complexity, operational costs and maintenance.. Think of it as getting ~80% of Kafka’s benefits with ~20% of the operational cost. Take a look at the POC repo.

Here is the new compute logs and events architecture proposition. A new custom component was needed in place of FluentBit: Log Forwarder. This was to be able to implement missing resilience properties, custom logs transformations to fit Redis Streams and our particular use case that watches two parallel streaming pipelines from the same job with end-of-stream (EOS) signal.. The new custom service was necessary once Kafka was out of the picture.

flowchart LR

%% =========================
%% SOURCES
%% =========================
subgraph S[Compute Runtime]
    K8S[Kubernetes Jobs / Pods]
    DCKR[Docker Containers]
end

%% =========================
%% EVENT PIPELINE
%% =========================
subgraph E[Event Pipeline]
    EF[Event Forwarder Daemon - async watcher process]
    K1[(Redis Streams: job-status-events)]
    ST[Status Consumer Service - async worker process]
    Q[Task Queue - Celery / RQ]
end

%% =========================
%% LOG PIPELINE
%% =========================
subgraph L[Log Pipeline]
    LF[Log Forwarder Daemon - async watcher process]
    K2[(Redis Streams: job-logs)]
    ES[(OpenSearch)]
end

%% =========================
%% REAL-TIME STREAMING LAYER
%% =========================
subgraph R[Real-time Log Streaming Layer]
    LC[Log Consumer Service - async worker process]
    RD[(Redis Pub/Sub)]
    DJ[SSE Service - auth + streaming]
    UI[Web UI - log viewer]
end

%% =========================
%% EVENT FLOWS
%% =========================
K8S --> EF
DCKR --> EF
EF --> K1
K1 --> ST
ST --> Q

%% =========================
%% LOG FLOWS
%% =========================
K8S --> LF
DCKR --> LF
LF --> K2
K2 --> LC
LC -->  ES

%% =========================
%% STREAMING FLOWS
%% =========================
LC --> RD
RD --> DJ
DJ --> UI

Loading

Resilience properties with Redis Streams:

Property	Event Forwarder	Log Forwarder	Status Consumer	Log Consumer
Stateless	Yes — no local state	Yes — no local state	Yes — state in Celery/PostgreSQL	Yes — state in OpenSearch
Restart-safe	Re-lists all containers on startup	Re-attaches to running container log streams	Resumes from PEL via XAUTOCLAIM	Resumes from PEL via XAUTOCLAIM
Reconnects	Exponential backoff on Docker/K8s stream disconnect	Exponential backoff on docker/k8s log stream disconnect	redis-py async reconnect	redis-py async reconnect
Dedup	Deterministic event_id + LRU cache	Deterministic event_id per line	PostgreSQL ON CONFLICT upsert (via Celery Worker)	OpenSearch document indexing
Backpressure	Bounded streams via MAXLEN ~	Bounded streams via MAXLEN ~	Per-shard batch reads with explicit XACK	Batched processing with configurable flush
Dead letters	—	—	stream:job-status-dlq after N deliveries	stream:job-logs-dlq after N deliveries
Horizontal scale	Single writer (by design; idempotent on restart)	Single forwarder per compute runtime	Add replicas → each acquires a shard lease	Add replicas → each acquires a shard lease

Durability and HA:

• Single-node Redis (default): appendonly yes + appendfsync everysec is configured everywhere Redis runs — the docker-compose redis service, kubernetes/20-infra/redis.yaml, and the integration-test Redis in kubernetes/tests/integration-stack.yaml. This bounds data loss to roughly one second of writes on an fsync gap.
• Sentinel HA (opt-in): kubernetes/20-infra/redis-ha.yaml ships a 3-node replicated Redis StatefulSet with a separate 3-node Sentinel StatefulSet. Workers opt in by setting REDIS_URL=redis+sentinel://redis-sentinel:26379/mymaster/0. The create_redis_client() factory in chris_streaming/common/redis_stream.py detects the redis+sentinel:// scheme and resolves the current master via redis.asyncio.sentinel.Sentinel, so no client code changes are needed to switch between single-node and Sentinel deployments.

[jennydaman]

I like where this is going in terms of simplifying the architecture. But something which makes me hesitant about LLM-generated solutions and architecture design are that LLMs are fundamentally restricted to generating text about outdated software and even outdated concepts. Right off the bat, I must nit-pick your code for using the redis:7-alpine (and postgres:16-alpine) image which has been unsupported for a year already. Besides, you don't mention Valkey which the community tends to recommend over Redis—but this news is only 2 years old, your LLM might not have seen enough of it in its training dataset yet.

This is a PoC so I should give it some slack but if we are trying to architect something new and innovative, we should be looking forwards to the future not backwards in time. While you could try to bias an LLM using a prompt telling it to do more web searches, this might fix the outdated version numbers but it won't make it capable of actual thinking nor research.

[jennydaman]

It would be rude of me to provide a blanket statement of LLM hate without offering anything else of substance. Well, here is a proposal for what I have in mind in terms of something as simple as possible while being reasonably scalable/performant:

flowchart LR

%% =========================
%% SOURCES
%% =========================
subgraph S[Compute Runtime]
    K8S[Kubernetes Jobs / Pods]
    DCKR[Docker Containers]
end

%% =========================
%% EVENT PIPELINE
%% =========================
subgraph E[Event Pipeline]
    EF[Event Forwarder Daemon - async watcher process]
    PG[PostgreSQL]
end

%% =========================
%% LOG PIPELINE
%% =========================
subgraph L[Log Pipeline]
    FB[Fluent Bit / Vector - log agent]
    ES[(Loki)]
end

%% =========================
%% REAL-TIME STREAMING LAYER
%% =========================
subgraph R[Real-time Log Streaming Layer]
    DJ[Django SSE Service - auth + streaming]
    UI[Web UI - log viewer]
end

%% =========================
%% EVENT FLOWS
%% =========================
K8S --> EF
DCKR --> EF
EF --> PG
PG --> DJ

%% =========================
%% LOG FLOWS
%% =========================
K8S --> FB
DCKR --> FB
FB --> ES

%% =========================
%% STREAMING FLOWS
%% =========================

ES --> DJ
DJ --> UI

Loading

• FluentBit/Vector writes directly to Loki. Streaming to front-end reads directly from Loki.
• "Event Forwarder" writes directly to PostgreSQL¹.
• "Middle man" is removed, whether it be Kafka or Redis.
• This is acceptable at small-medium scale, e.g. it could easily probably handle 1GiB/s².

Footnotes

1. https://mccue.dev/pages/8-16-24-just-use-postgres#why-not-kafka ↩

2. In our Landmark 6th floor server room, some computers are connected to hubs, some are using cat5e, some are cat6. cat5e goes up to 119MiB/s, cat6 can do 1.2GiB/s. That is to say, before we think about the horizontal scalability of our software design, we need to solve the bottleneck that is our (lack of) networking hardware. ↩

[jennydaman]

Log and Event Collector

FluentBit v.s. Vector v.s. OpenTelemetry

I strongly recommend the OpenTelemetry Collector (otel-collector) over FluentBit. Vector is a great piece of software but its community seems to be shrinking as people are standardizing around otel-collector and migrating to it. In fact, both Vector and FluentBit support OpenTelemetry protocols in order to keep up with the industry's standardization around OpenTelemetry. This is useful for existing systems with legacy architectures that depend on Vector or FluentBit, but we are trying to design and implement a new architecture so... Might as well choose the OpenTelemetry and forget about what are being replaced by it.

To my knowledge, FluentD (not FluentBit) played an important role in recent history circa 2014–2020 because it was the only good option at the time, hence it has built-in support with docker. FluentBit came after and it does not have the same special treatment from docker because nowadays the ecosystem has expanded so shipping built-in drivers for any one component makes less sense.

It seems like copy-pasting from a chatbot is what's cool, so here is what GPT-5 mini with reasoning + web search has to say:

PROMPT: Please compare the OpenTelemetry Collector, FluentBit, and Vector. Make a suggestion for which one to use in a new software architecture for log and event streaming on a Kubernetes cluster. Use the most up-to-date recommendations and advice from the years 2025 and 2026.

Searching for recent facts

I need to conduct a WebSearch for real-world facts and recent information. The rules specify that I should use the related search terms at the end and follow the proper citation format. Today’s date is April 19, 2026, so I’ll make sure to incorporate that. I’ll aim for a concise comparison and suggestions in the response while using headings for organization. Lastly, I’ll include citations at the end of each paragraph. Let’s get started!Comparing sources for benchmarks

I see that I have enough sources lined up for benchmarks, including ones from VictoriaMetrics, Parseable, SigNoz, and Atatus from different years. It would be wise to also compare with Vector. So, I need to search for updates related to Vector from 2025-26. This should help me gather a comprehensive view for comparison. Let's get into it!Creating comparison for log streaming

I’m gathering resources on Vector releases and benchmarks to craft a concise comparison. I'll cover strengths, weaknesses, resource usage, and community recommendations. For a new Kubernetes cluster needing logs and event streaming, I recommend OpenTelemetry Collector as the central pipeline. Fluent Bit is a lightweight option for forwarding logs, while Vector offers powerful transforms and good performance—though licensing concerns might arise. I’ll suggest a hybrid approach! Lastly, I’ll invoke the RelatedSearchTerms tool with relevant suggestions.## Short comparison (2025–2026)

• OpenTelemetry Collector

  • Strengths: Vendor‑neutral unified MELT (metrics, events/logs, traces) pipeline, native OTLP, rich processors (OTTL), good for central aggregation/gateway and advanced sampling/enrichment. Best for correlating logs+traces+metrics and for multi‑backend routing.
  • Trade-offs: Larger footprint (contrib builds), higher memory/CPU than tiny shippers; historically weaker local persistent buffering (persistent queue matured recently).

• Fluent Bit

  • Strengths: Extremely lightweight C binary ideal as a per‑node DaemonSet for tailing container stdout/files; mature tailing, filesystem buffering, tiny memory use, many outputs/plugins, battle‑tested at high throughput. Excellent at edge/resource‑constrained nodes.
  • Trade-offs: Focused on logs (less on metrics/traces), less powerful native transform language than Vector/OTTL; sometimes requires pairing with a central collector for MELT needs.

• Vector

  • Strengths: Modern, high‑performance Rust implementation with expressive VRL transform language, strong observability of its own components, growing sink/source ecosystem, active 2025–2026 feature cadence (OTLP support, buffer metrics, improved sinks). Good balance of transform power and performance.
  • Trade-offs: Heavier than Fluent Bit (but usually lighter than full OTEL contrib builds), historically had edge cases in file‑rotation/backpressure handling in some benchmarks; performance varies by configuration.

Performance snapshot (real‑world benchmark summary March 2026): VictoriaMetrics’ benchmark showed Fluent Bit and Vector both outperforming the OpenTelemetry Collector for raw log throughput in their test, with Fluent Bit typically showing the best CPU efficiency and Vector close behind; OTEL Collector prioritized multi‑signal processing over raw max throughput.

Practical recommendation for a new Kubernetes log & event streaming architecture (2026)

• Default (recommended): Hybrid agent + gateway

  • Deploy Fluent Bit as a lightweight DaemonSet on every node to tail container stdout, add Kubernetes enrichment, local filesystem buffering for resiliency, and forward via OTLP/HTTP to a centrally managed OpenTelemetry Collector (as gateway). Use the Collector (gateway or an autoscaled deployment) for durable processing, OTTL transforms, sampling, and routing to multiple backends (logs store, metrics backend, tracing backend). Rationale: minimal node footprint, proven tailing, unified MELT processing & vendor neutrality at the gateway.

• Alternative (single‑tool) when you want richer local transforms and a modern Rust stack:

  • Use Vector as the node agent (DaemonSet) and/or a central gateway: it supports OTLP, powerful VRL transforms, and efficient sinks; choose Vector if you prefer a single consistent config language for transforms and tighter control of per‑event processing at the edge. Caveat: validate buffering/rotation behavior for your workload and test restart/backlog scenarios.

• When to choose pure OpenTelemetry Collector only:

  • If your primary goal is full MELT correlation out of the box and you can accept higher per‑node resource use (or run the Collector as an agent with careful sizing), or if you want to avoid protocol translations and keep strict OTLP end‑to‑end. Otherwise, pair it with a lightweight shipper for better node efficiency.

Quick deployment checklist (defaults you can use)

1. Fluent Bit DaemonSet (tail + kubernetes filter + filesystem buffer) -> OTLP exporter to:
2. OpenTelemetry Collector (gateway/service) with receivers: otlp; processors: k8sattributes, batch, OTTL transforms; exporters: log store(s)/trace/metrics. Test persistent queue settings and resource requests/limits.

Final notes

• Benchmarks and feature sets changed in 2025–2026; run a short pilot with representative log volumes and failure scenarios (rotate, restart, network loss) before full rollout. VictoriaMetrics’ 2026 benchmarks are a useful reference for throughput/resilience tradeoffs.

Related searches you might find useful: OpenTelemetry Collector vs Fluent Bit 2026 comparison; Vector v0.53 2026 features and benchmarks; Kubernetes logging best practices 2026 Fluent Bit OpenTelemetry Vector.

My response: this demonstrates the capabilities and lack thereof of using ChatGPT for research and technical decision-making. The LLM performed a web search and found benchmarks showing that FluentBit and Vector are more resource-efficient than otel-collector, and it does not recommend using only otel-collector unless "you can accept higher per‑node resource use." This all makes sense since FluentBit is written in C, Vector in Rust, and otel-collector in Go. However, ChatGPT has never had to deploy on Kubernetes clusters, to be held accountable for its actions, and to endure all-nighters when things go wrong and what it thought was easy actually turns out to be hard. I have.

Experienced human senior developers all agree:

• Making decisions solely based on benchmarks is dumb.
• Avoid complexity at all costs.

ChatGPT's "recommended" "hybrid" solution is a terrible idea. Using two components is twice the amount of work compared to having one. Its recommendations to use FluentBit or Vector over otel-collector make sense for resource-constrained environments, but the general trend in industry is that developer time costs way more than CPU time. (I could keep beating this dead horse... a common log data transformation is to filter out unimportant lines, so performing OTTL transformations on the gateway instead of locally within DaemonSet containers means wasteful network I/O, which is even slower and more expensive than CPU time. ChatGPT speaks of this in saying vector can do "per‑event processing at the edge" yet it's unable to approximate nor judge how important edge-processing actually is.)

[jbernal0019]

@jennydaman We need the "middle man", Redis Streams in this case. There are many reasons for that (eg. resilience properties above) but here are three very important:

1- We can tolerate loosing some logs but can not tolerate losing events. Writing directly to Postgres from Event Forwarder is a terrible solution. It will also put more pressure on Postgres DB and doesn't fit our CUBE implementation. All the events processing and pfcon orchestration is done through Celery workers.
2- Postgres and Loki are in a different network partition on the consumers side. Auth layer and communication surface is also provided by Readis Streams.
3- We need a custom logs forwarder because of the use of Readis Streams and how CUBE's worker orchestrates job lifecycle through pfcon. There is logic there (EOS signal) to prevent containers from being deleted before logs are fully flushed.

Also I think the system above makes a good compromise between simplicity and easier to swap later for a higher performant system based on Kafka. I would remove Redis Pub/Sub from the picture but the SSE server then becomes more complex. It would no longer be stateless (easier to scale horizontally) and need to handle:

• multiple clients
• filtering per client
• distributing messages
• other non-interesting stuff

[jennydaman]

Losing Events is Not Solved by a Middle Man

... It is solved by high-availability and/or some simple programming.

What magic does a "middle man" provide which solves the problem of losing events? You indicate that Redis could run either in single-node mode, where the problem is bound "to one second of writes on an fsync gap," or run Redis in HA mode. If you're talking about high-availability (HA) mode, well... If we have an HA k8s API-server and HA PostgreSQL, we won't lose events (within what's possible as per CAP theorem).

We wouldn't be losing events in single-node mode either, with or without a "middle man." Remember that the k8s API-server is comprised of a database (etcd) and we're essentially trying to copy data out of one database (k8s API-server's etcd) to another (CUBE's PostgreSQL). There is persistence on each end of the pipeline. An events forwarding pipeline can be made resilient to downtime with a simple mechanism:

# Python-like pseudocode, function names are made up

# Ask k8s, how long have I been dead?
previous_instance = k8s_client.get_events(pod=myself)

# Get missed events between time when I died and now
missed_events = k8s_client.get_events(from=previous_instance.time_when_stopped, to=time.now())

# Store missed events
cube_pg.receive(missed_events)

# Listen for current events and stream to CUBE
for event in k8s_client.watch():
    cube_pg.receive(event)

It will also put more pressure on Postgres DB

Yes, but a middle man wouldn't help.

Again, what magic does a "middle man" provide which reduces pressure on the Postgres DB? A middle man could optimize the writes by buffering them, but one way or another, if k8s has 10 events than Postgres needs to write 10 events—a middle man doesn't magically reduce the pressure.

Buffering is a good idea, which is why it's a built-in feature of OpenTelemetry-Collector. See https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor

Postgres and Loki are in a different network partition on the consumers side.

I don't know what you mean by this. We are the ones designing the architecture. We can choose how to partition the network. We could choose to not partition the network at all.

Auth layer and communication surface is also provided by Readis Streams.

Redis, as a commercial company, tries to pack their product with features so that they can cast a wide net upon potential customers.

The state-of-the-art in east-west network security is to use, as you mentioned, network partitioning (configuration of firewall-like rules to only allow services to communicate among architectural dependency relationships and deny networking to unrelated pods) and/or mutual TLS (mTLS)¹. Authentication would add a redundant security check and configuration. It's not a big selling point when trying to choose a component intended for cloud-native deployment.

EOS Signal to prevent containers from being deleted

Containers are never deleted from the Kubernetes API unless you explicitly delete them or have created a controller which deletes them. I don't see your point, neither do I see how come a "custom logs forwarder" is required, and why our current architecture is supposedly dependent on the future introduction of specifically Redis Streams.

1. pfcon receives logs and events.
2. wait for event indicating that containers have exited.
3. check or wait for indication that logs are flushed.
4. pfcon sends request to delete pod.

The process is simple in my mind. Am I missing something?

To Be Clear...

I support the idea of using Redis (or Valkey, or alternatives) as a middle man, but I am pushing back on it because only good ideas can withstand scrutiny. I want to verify that we all understand the necessity of some "middle man" and that we're not doing pre-mature optimization based on last generation's designs.

Project Management

@rudolphpienaar in addition to the questions about requirements I posed in #151 (comment), you as a manager also need to decide: this discussion proposes a big leap forward. Waterfall or agile development?

Footnotes

1. https://www.buoyant.io/mtls-guide ↩

[jbernal0019]

@jennydaman Great that you support the idea of using Redis. I don't think anybody needs to be convinced about why it's so important to decouple producers from consumers with an appropriate message queue system (helping with resilient features) in an event-driven architecture.

Your comment "k8s API-server is comprised of a database (etcd) and we're essentially trying to copy data out of one database (k8s API-server's etcd) to another (CUBE's PostgreSQL)". This is an oversimplification of the whole ChRIS system. The whole plugin lifecycle, related event processing and PostgreSQL database schema and interactions are orchestrated from the client side (CUBE's Celery worker). pfcon doesn't do anything by itself. It's just a stateless service driven from the client side. As part of the orchestration, containers are deleted from Docker/Kuberrnetes when a terminal status is reached. We need to implement the EOS signal to avoid losing logs because of premature container deletion.

We don't want to couple producers and consumers through data stores. Also PostgreSQL and Loki doing all the fan out work to consumers, filtering per client, distributing messages and pushing to the SSE event service in your proposed architecture looks really strange :-)

The network partitioning was the first and only thing @rudolphpienaar asked and cared about when I presented the Kafka-based architecture. And I agree we should keep that in mind and also strive to minimize the communication surface and auth setup between the services running on the remote compute cluster and CUBE's consumers.

Each arch has different trade offs and put complexity in different places and again I think the system I'm proposing not only fits better with current ChRIS implementation but also makes a good compromise between simplicity and easier to swap later for a higher performant system based on Kafka when that moment comes if ever.

The weakest point in my arch above is the Log Consumer. Needs to be scalable. Kafka was able to write directly to OpenSearch but Redis Streams is not.

[chhsiao1981]

@jennydaman

1. please do not use meme pictures in the discussions. The meme pictures are usually:
   • implying sarcastic comments to others.
   • distracting from the topic-related content.
   • wasting time in generating these pictures.
2. I believe the "logs" in this discussion is about the "logs" generated from the "ChRIS apps" that are printed to stdout/stderr if the apps are executed in terminal. Such kind of logs are stored in the specific files in container daemons and are irrelevant to etcd logs.
3. Your figure indicates that the SSE-server (@jbernal0019 already removed "django" in this block) will get the data directly from Postgres / loki. However, as far as I know, there is no event-watcher api for loki. Therefore, my understanding of loki => SSE-server is that SSE-server needs to periodically query the results from loki. @jbernal0019 's redis pub/sub => SSE-server setup allows event-driven approach for SSE-server to subscribe from redis and wait for the new updates of the keys.

[chhsiao1981]

My understanding of loki is that it mainly replaces elasticsearch/opensearch to be an alternative lightweight persistent storage for log. The "lightweight" comes from that loki indexes based on "labels", not the "full text".

[jennydaman]

You're right about logs being stored in the "specific files in container daemons" and not in etcd. Regardless, they can be streamed from the Kubernetes API server, from the container daemons, or from those files directly.

You're also correct about Loki not having an event-watcher API. The design I am suggesting is modeled after how OpenShift works, meaning it is an architecture proven to work for enterprise customers. Importantly, we must recognize that log streaming and long-term storage are two separate concerns.

• All logs must be handled by long-term storage.
• A tiny subset of logs need to be streamed to the front-end.

OpenShift solves this problem by streaming logs to the front-end on-demand, and only when requested. In the background, all logs are being collected and sent to LokiStack.

In ChRIS, we could mimic OpenShift's approach, or we could have the front-end poll Loki. Does polling Loki satisfy our requirements for being able to view logs of running plugin instances from the front-end? Maybe. We need to clarify what our requirements are first.

FWIW I mention Loki only as an example, because it's the most popular option in the community. In my diagram you could substitute Loki for OpenSearch, QuickWit, or whatever you want.

[chhsiao1981]

hi @jennydaman ～

Thanks for the clarification～

I believe Openshift's approach, as streaming by directly quasi-tail -f the logs to the front-end on-demand through SSE, would be the easiest implementation. However, the caveats:

• The plugin-instances are usually long-running apps (6+ hours) and can generate millions of lines. Do we do only the tail -f style that shows only the n newest lines?
• Do we want to keep all the millions of lines of the logs? or the purpose is simply only to let the users "feel" that the plugin-instances are still working-in-progress? (I would vote for all the millions of lines of the logs in the long-term storage for future debugging. but unsure about the streaming part.)

[jennydaman]

OpenShift Console (OCP) is just a Kubernetes front-end. The k8s API is flexible, figuring out how to use it is a front-end concern. Here is a suggested front-end implementation plan:

• MVP: front-end simply runs the equivalent of tail -f or kubectl logs -f
• Stage 2: front-end evicts older lines of log and imposes a cap on how much memory is used by the log buffer.
• Stage 3: when user scrolls up, cancel the "follow/watch" stream and switch to making short HTTP requests to receive log chunks. This works like a cursor, e.g. when you scroll up, from JS a function does the equivalent of kubectl logs --since=1m. Scroll up again and the JS function calls kubectl logs --since=2m. When the user scrolls back all the way down, resume the "follow/watch" stream.

[jbernal0019]

@jennydaman I have a doubt about Loki. Everywhere I search about it seems to imply that it's designed to work better with object storage (S3).

I asked AI to compare Loki and OpenSearch for our use case and POC code. This is what it said:

What OpenSearch actually does today

The write and read paths use OpenSearch for:

1. Bulk write of batches with daily index rotation.
2. Exact point lookup on job_id (term query), sorted timestamp:asc, size up to 5000 — for SSE historical replay.
3. One public JSON endpoint (/logs/{job_id}/history) using the same term query with pagination.

No match, no full-text, no aggregations, no wildcards, no scripts. OpenSearch is used as a time-sorted, keyword-indexed KV store keyed by job_id — nothing more.

How Loki's data model fits that

Loki indexes logs by labels only; content sits in compressed chunks on object storage. Two consequences matter:

• job_id cannot be a label. It's inherently high-cardinality (one stream per job would be pathological — Loki's docs are explicit about that). In Loki v3 you'd put job_id in structured_metadata, which means queries become {service="chris-logs", namespace="..."} | job_id="chris-jid-42" — a chunk scan within the label-selected streams, not a direct lookup.
• Queries require a time window. OpenSearch's current term query with size: 5000 has no time bound. Loki's LogQL needs start/end. You'd derive the window from job_workflow.created_at (or use a generous default like 7d). Doable, but it's a new coupling.

Pros / cons for this use case

Pros of Loki

• Cost: 10–50× cheaper storage on high-volume pipelines (object storage + compressed chunks vs. OpenSearch replicas).
• Ops: single-binary mode is much simpler than OpenSearch cluster management; no shard rebalancing, no JVM tuning.
• Retention per label stream, native S3/GCS lifecycle.

Cons of Loki for this access pattern

• The query is "give me every line for job_id=X, sorted by time." Loki will have to fetch and scan chunks matching the label selector within the time window. For hot chunks (recent jobs) this is fast; for old ones it's IO-bound on object storage.
• No point-in-time dedup via a keyword field. The event_id dedup (between replayed-history and live Pub/Sub) would have to be applied on the client side after full chunk scan — same as today, but potentially over more data.
• Loki's push API does not natively respect the EOS flush gate semantics; the "flush then SET key" logic stays under app control, so that's a wash.

Recommendation

Not a clear win unless log volume is your cost driver. The access pattern here is "point-read by job_id within a bounded time window" — OpenSearch's inverted index on job_id (a keyword field) handles that natively; Loki makes you scan. If you're pushing less than ~50 GB/day of logs, OpenSearch is cheaper in engineering terms than the Loki migration. If you're pushing hundreds of GB/day and the cluster is painful, Loki + structured_metadata + a time-window derived from job_workflow.created_at is a valid target — you trade some query latency for dramatically less storage cost and ops.

[jennydaman]

Yes, I don't particularly recommend Loki. I mention it because (a) it's popular, (b) integrated with OpenShift.

Loki greatly benefits from S3 storage. But so does OpenSearch? Any time you want horizontal scaling and storage of arbitrarily large, unstructured data, object storage is implicated.

The performance penalty of Loki and high-cardinality labels is a good point and I'm glad you mention it. It's exactly why I advocate for QuickWit or OpenObserve instead.

Between QuickWit and OpenSearch:

• OpenSearch: advanced features for full-text and semantic search, industry standard and battle-tested
• QuickWit: orders of magnitude better performance and lower costs, simpler mode of operation.

[jbernal0019]

@jennydaman @chhsiao1981 @rudolphpienaar @Sandip117

I've slimmed down the POC by now using Quickwit instead of OpenSearch and dropping Redis pub/sub. The SSE server now directly pulls events and logs for the near-realtime path from Redis Streams. It's a bit more complex now as it does the fan out work that was previously done by Redis pub/sub. This is supposed to be enough for hundreds of concurrent viewers.
The Log Consumer now becomes just a connector between Redis Streams and Quickwit. The Status Consumer is just a bridge between Redis Streams and Celery worker.

I think this is a minimalistic architecture that's easy to reason about, supports a moderate scale of UI streams and fits well with current ChRIS system. Here is the new architecture diagram. It's a bit simplified because the SSE service actually replays historical logs and events to late UI users from Quickwit and PostgreSQL This is in addition to the live stream if it's still happening when user connects.

flowchart LR

%% =========================
%% SOURCES
%% =========================
subgraph S[Compute Runtime]
    K8S[Kubernetes Jobs / Pods]
    DCKR[Docker Containers]
end

%% =========================
%% EVENT PIPELINE
%% =========================
subgraph E[Event Pipeline]
    EF[Event Forwarder Daemon - async watcher process]
    K1[(Redis Streams: job-status-events)]
    ST[Status Consumer Service - async worker process]
    Q[Task Queue - Celery / Redis]
    P[(PostgresSQL)]
end

%% =========================
%% REAL-TIME STREAMING LAYER
%% =========================
subgraph R[Real-time Streaming Layer]
    DJ[SSE Service - auth + streaming]
    UI[Web UI - log/event viewer]
end

%% =========================
%% LOG PIPELINE
%% =========================
subgraph L[Log Pipeline]
    LF[Log Forwarder Daemon - async watcher process]
    K2[(Redis Streams: job-logs)]
    LC[Log Consumer Service - async worker process]
    ES[(Quickwit)]
end

%% =========================
%% EVENT FLOWS
%% =========================
K8S --> EF
DCKR --> EF
EF --> K1
K1 --> ST
ST --> Q
Q -->  P

%% =========================
%% LOG FLOWS
%% =========================
K8S --> LF
DCKR --> LF
LF --> K2
K2 --> LC
LC --> ES

%% =========================
%% STREAMING FLOWS
%% =========================
K1 --> DJ
K2 --> DJ
DJ --> UI

Loading

[chhsiao1981]

@jbernal0019

The caveat with pure-redis based streaming is that redis is in-memory only storage.
We don't have abundant mem for big-data / 100k+ logs of plugin-instances.

[chhsiao1981]

@jennydaman

The caveat of Openshift's direct-streaming-log on-demand approach:

1. It's target audience is for system-admin, which is around 10~20 people in a company / production use. However, our target audience is our users, which is foreseeable 1k+ people for each hospital, and 100k+ people for the hub.
2. We do want to have the functionality that the logs can be retrieved based on time-range, so we don't need to waste resource always pulling the full logs when retrieving the logs. direct-streaming-log on-demand does not fit this needs.

[jennydaman]

1. You're right, I am putting out ideas which consider small-medium scale and reasonable goals for agile development, while avoiding premature optimization. There is no point to developing a system which can handle 100k+ when we lack both demand and capacity. As I noted above: for our internal ChRIS, we'd need to upgrade our Ethernet cables and spin out dedicated networking hardware in order to handle 1k+ concurrent users streaming fast-paced logs. If we hope to scale up to 100k+ people on the hub some day, I don't even know what kind of computing infrastructure we would need for that. I do know it'll require much more work including some major potentially backwards incompatible redesign to fundamentals of how CUBE works (e.g. database design¹). Again it goes back to the question, what is the scope of this discussion? Are we trying to implement log streaming and events at a small-medium scale? Are we trying to redesign the entire ChRIS in a multi-year project to handle large-national scale?
2. As aforementioned, logs can be retrieved based on time range (without pulling full logs) from k8s API (or Loki, QuickWit). The k8s API design is not perfectly convenient but it is definitely possible to retrieve based on time range using the since_seconds and limit_bytes².

Footnotes

1. In 2011, Instagram was a Django + Postgres application which famously scaled up to 14 million users. Their engineering blog provides insight into solutions which we should take as inspiration. https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c ↩

2. https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/CoreV1Api.md#parameters-162 ↩

[chhsiao1981]

1. I believe Red Hat will take care of the hardware deployment / openshift setup for the 100k+ users on the Hub.
2. The way that "Since" works: read the log files, return only the msg.timestamp >= Since. https://github.com/moby/moby/blob/master/daemon/logger/loggerutils/logfile.go#L886
3. Thanks for the info about "Since". I wasn't aware of such functionality in docker/k8s. Turns out docker/moby logs the stdout/stderr logs in json-format, with {"log", "stream", "time"}
4. I believe there is one caveat: retrieving based on time-range vs. retrieving in streaming. As what @jennydaman pointed out, the frontend should be aware of whether the UI is in the end of log or not.
   • If the UI is in the end of the log, then the frontend should present quasi-tail -n 100-like info. Since docker/moby logs is with "time", the frontend should also be aware of the "time" of the 1st line.
   • If the UI is not in the end of the log, the frontend should be able to know the range of the time to retrieve, and retrieve from loki (I would prefer loki at this moment).
   • The UI should be able to smoothly switch between "the-end/not-the-end of the log" based on the "time" in the log.
5. As for the streaming part, I believe we can live with "direct-using k8s-api" for now, and have more advanced queuing-streaming mechanism in the future if we do hit some performance issue.

[chhsiao1981]

@jennydaman

I saw you have Instagram's reference about sharding on Postgres for 14M+ users.

1. sharding is a very typical method for big-data, keywords including consistent hashing / distributed hash table https://en.wikipedia.org/wiki/Shard_(database_architecture).
2. One of the many reasons why MongoDB is popular nowadays (in addition to using json instead of SQL as the interfacing language), is because MongoDB fundamentally solves many issues about sharding / distributed database (sharding management / auto-resharding / routing for sharding).

[jennydaman]

https://iggy.apache.org/ is an interesting upcoming Kafka competitor.