Skip to content
/ vector Public
forked from vectordotdev/vector

A lightweight and ultra-fast tool for building observability pipelines

License

Notifications You must be signed in to change notification settings

apestel/vector

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Components  •  Guides  •  Docs  •  Blog  •  Community  •  Download v0.9.1


Vector


New post! How We Test Vector


What is Vector?

Vector is a lightweight, ultra-fast, open-source tool for building observability pipelines. Compared to Logstash and friends, Vector improves throughput by ~10X while significanly reducing CPU and memory usage.

Principles

  • Reliability First. - Built in Rust, Vector's primary design goal is reliability.
  • One Tool. All Data. - One simple tool gets your logs, metrics, and traces (coming soon) from A to B.
  • Single Responsibility. - Vector is a data router, it does not plan to become a distributed processing framework.

Who should use Vector?

  • You SHOULD use Vector to replace Logstash, Fluent*, Telegraf, Beats, or similar tools.
  • You SHOULD use Vector as a daemon or sidecar.
  • You SHOULD use Vector as a Kafka consumer/producer for observability data.
  • You SHOULD use Vector in resource constrained environments (such as devices).
  • You SHOULD NOT use Vector if you need an advanced distributed stream processing framework.
  • You SHOULD NOT use Vector to replace Kafka. Vector is designed to work with Kafka!
  • You SHOULD NOT use Vector for non-observability data such as analytics data.

Community

  • Vector is downloaded over 100,000 times per day.
  • Vector's largest user processes over 10TB daily.
  • Vector is used by multiple fortune 500 companies with stringent production requirements.
  • Vector has over 15 active contributors and growing.

About

Setup

Reference

Administration

Resources

Performance

The following performance tests demonstrate baseline performance between common protocols with the exception of the Regex Parsing test.

Test Vector Filebeat FluentBit FluentD Logstash SplunkUF SplunkHF
TCP to Blackhole 86mib/s n/a 64.4mib/s 27.7mib/s 40.6mib/s n/a n/a
File to TCP 76.7mib/s 7.8mib/s 35mib/s 26.1mib/s 3.1mib/s 40.1mib/s 39mib/s
Regex Parsing 13.2mib/s n/a 20.5mib/s 2.6mib/s 4.6mib/s n/a 7.8mib/s
TCP to HTTP 26.7mib/s n/a 19.6mib/s <1mib/s 2.7mib/s n/a n/a
TCP to TCP 69.9mib/s 5mib/s 67.1mib/s 3.9mib/s 10mib/s 70.4mib/s 7.6mib/s

To learn more about our performance tests, please see the Vector test harness.

Correctness

The following correctness tests are not exhaustive, but they demonstrate fundamental differences in quality and attention to detail:

Test Vector Filebeat FluentBit FluentD Logstash Splunk UF Splunk HF
Disk Buffer Persistence ⚠️
File Rotate (create)
File Rotate (copytruncate)
File Truncation
Process (SIGHUP) ⚠️
JSON (wrapped)

To learn more about our correctness tests, please see the Vector test harness.

The Little Details

Data Model

Control Flow

Data Processing

Operations

Reliability

  • Memory safety - Vector is built in Rust and is memory safe, avoiding a large class of memory related errors.
  • Decoupled buffer design - Buffers are per-sink; a bad sink won't bring the entire pipeline to a halt.
  • Intelligent retries - A fibonacci backoff algorithsm with jitter makes Vector a good citizen during outages.
  • Backpressure & load shedding - Buffers can be configured to provide backpressure or shed load.
  • Rate-limited internal logging - Vector's internal logging is rate-limited avoiding IO saturation if errors occur.
  • Sink healthchecks - Healthchecks provide startup safety and prevent deploys with bad configuration.
  • Robust disk buffering - Vector uses leveldb for robust data durability across restarts.

UX

Installation

Run the following in your terminal, then follow the on-screen instructions.

curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | sh

Or use your own preferred method.

Latest Highlights

view all...


Developed with ❤️ by Timber.io - Security Policy - Privacy Policy

About

A lightweight and ultra-fast tool for building observability pipelines

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Rust 90.0%
  • HTML 10.0%