Merge pull request #2225 from fluent/alexakreizinger/sc-153842/consolidate-unit-sizes-into-configure-fluent

Author: alexakreizinger

.gitbook.yaml

@@ -101,3 +101,4 @@ redirects:
     about/sandbox-and-lab-resources: ./about/resources.md
     installation/downloads/amazon-ec2: ./installation/downloads/linux/amazon-linux.md
     administration/configuring-fluent-bit/yaml/configuration-file: ./administration/configuring-fluent-bit/yaml.md
+    administration/configuring-fluent-bit/unit-sizes: ./administration/configuring-fluent-bit.md#unit-sizes

SUMMARY.md

@@ -38,7 +38,7 @@
 ## Administration
 
 * [Configure Fluent Bit](administration/configuring-fluent-bit.md)
-  * [YAML configuration](administration/configuring-fluent-bit/yaml.md)
+  * [YAML configuration files](administration/configuring-fluent-bit/yaml.md)
     * [Environment variables](administration/configuring-fluent-bit/yaml/environment-variables-section.md)
     * [Includes](administration/configuring-fluent-bit/yaml/includes-section.md)
     * [Service](administration/configuring-fluent-bit/yaml/service-section.md)
@@ -47,15 +47,14 @@
     * [Pipeline](administration/configuring-fluent-bit/yaml/pipeline-section.md)
     * [Plugins](administration/configuring-fluent-bit/yaml/plugins-section.md)
     * [Upstream servers](administration/configuring-fluent-bit/yaml/upstream-servers-section.md)
-  * [Classic mode](administration/configuring-fluent-bit/classic-mode.md)
+  * [Classic configuration files](administration/configuring-fluent-bit/classic-mode.md)
     * [Configuration file](administration/configuring-fluent-bit/classic-mode/configuration-file.md)
     * [Commands](administration/configuring-fluent-bit/classic-mode/commands.md)
     * [Format and schema](administration/configuring-fluent-bit/classic-mode/format-schema.md)
     * [Record accessor syntax](administration/configuring-fluent-bit/classic-mode/record-accessor.md)
     * [Upstream servers](administration/configuring-fluent-bit/classic-mode/upstream-servers.md)
     * [Variables](administration/configuring-fluent-bit/classic-mode/variables.md)
   * [Multiline parsing](administration/configuring-fluent-bit/multiline-parsing.md)
-  * [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md)
 * [AWS credentials](administration/aws-credentials.md)
 * [Backpressure](administration/backpressure.md)
 * [Buffering and storage](administration/buffering-and-storage.md)

administration/configuring-fluent-bit.md

@@ -1,13 +1,26 @@
 # Configure Fluent Bit
 
-Fluent Bit supports two configuration formats:
+Fluent Bit uses configuration files to store information about your specified [inputs](../data-pipeline/inputs.md), [outputs](../data-pipeline/outputs.md), [filters](../data-pipeline/filters.md), and more. You can write these configuration files in one of these formats:
 
-- [YAML](./configuring-fluent-bit/yaml.md): Standard configuration format as of v3.2.
-- [Classic mode](./configuring-fluent-bit/classic-mode.md): To be deprecated at the end of 2026.
+- [YAML configuration files](./configuring-fluent-bit/yaml.md) are the standard configuration format as of Fluent Bit v3.2. They use the `.yaml` file extension.
+- [Classic configuration files](./configuring-fluent-bit/classic-mode.md) will be deprecated at the end of 2026. They use the `.conf` file extension.
+
+## Unit sizes
+
+Some configuration settings in Fluent Bit use standardized unit sizes to define data and storage limits. For example, the `buffer_chunk_size` and `buffer_max_size` parameters for the [Tail](../data-pipeline/inputs/tail.md) input plugin use unit sizes.
+
+The following table describes the unit sizes you can use and what they mean.
+
+| Suffix | Description | Example |
+| :--- | :--- | :--- |
+| _none_ | **Bytes**: If you specify an integer without a unit size, Fluent Bit interprets that value as a bytes representation. | `32000` means 32,000 bytes. |
+| `k`, `kb`, `K`, `KB` | **Kilobytes**: A unit of memory equal to 1,000 bytes. | `32k` means 32,000 bytes. |
+| `m`, `mb`, `M`, `MB` | **Megabytes**: A unit of memory equal to 1,000,000 bytes. | `32m` means 32,000,000 bytes. |
+| `g`, `gb`, `G`, `GB` | **Gigabytes**: A unit of memory equal to 1,000,000,000 bytes. | `32g` means 32,000,000,000 bytes. |
 
 ## Command line interface
 
-Fluent Bit exposes most of it features through the command line interface. Running the `-h` option you can get a list of the options available:
+Fluent Bit exposes most of its configuration features through the command line interface. Use the `-h` or `--help` flag to see a list of available options.
 
 ```shell
 # Podman container tooling.
@@ -16,167 +29,3 @@ podman run -rm -ti fluent/fluent-bit --help
 # Docker container tooling.
 docker run --rm -it fluent/fluent-bit --help
 ```
-
-Which returns the following help text:
-
-```text
-Usage: /fluent-bit/bin/fluent-bit [OPTION]
-
-Available Options
-  -b  --storage_path=PATH specify a storage buffering path
-  -c  --config=FILE       specify an optional configuration file
-  -d, --daemon            run Fluent Bit in background mode
-      --supervisor        run under a supervising parent process, uses "fork()" to execute child processes
-  -D, --dry-run           dry run
-  -f, --flush=SECONDS     flush timeout in seconds (default: 1)
-  -C, --custom=CUSTOM     enable a custom plugin
-  -i, --input=INPUT       set an input
-  -F  --filter=FILTER     set a filter
-  -m, --match=MATCH       set plugin match, same as '-p match=abc'
-  -o, --output=OUTPUT     set an output
-  -p, --prop="A=B"        set plugin configuration property
-  -R, --parser=FILE       specify a parser configuration file
-  -e, --plugin=FILE       load an external plugin (shared lib)
-  -l, --log_file=FILE     write log info to a file
-  -t, --tag=TAG           set plugin tag, same as '-p tag=abc'
-  -T, --sp-task=SQL       define a stream processor task
-  -v, --verbose           increase logging verbosity (default: info)
-  -Z, --enable-chunk-traceenable chunk tracing, it can be activated either through the http api or the command line
-  --trace-input           input to start tracing on startup.
-  --trace-output          output to use for tracing on startup.
-  --trace-output-property set a property for output tracing on startup.
-  --trace                 setup a trace pipeline on startup. Uses a single line, ie: "input=dummy.0 output=stdout output.format='json'"
-  -w, --workdir           set the working directory
-  -H, --http              enable monitoring HTTP server
-  -P, --port              set HTTP server TCP port (default: 2020)
-  -s, --coro_stack_size   set coroutines stack size in bytes (default: 24576)
-  -q, --quiet             quiet mode
-  -S, --sosreport         support report for Enterprise customers
-  -Y, --enable-hot-reload enable for hot reloading
-  -W, --disable-thread-safety-on-hot-reloadingdisable thread safety on hot reloading
-  -V, --version           show version number
-  -h, --help              print this help
-
-Inputs
-  blob                    Blob (binary) files
-  cpu                     CPU Usage
-  mem                     Memory Usage
-  thermal                 Thermal
-  kmsg                    Kernel Log Buffer
-  proc                    Check Process health
-  disk                    Diskstats
-  systemd                 Systemd (Journal) reader
-  netif                   Network Interface Usage
-  docker                  Docker containers metrics
-  docker_events           Docker events
-  podman_metrics          Podman metrics
-  process_exporter_metricsProcess Exporter Metrics (Prometheus Compatible)
-  gpu_metrics             GPU Metrics
-  node_exporter_metrics   Node Exporter Metrics (Prometheus Compatible)
-  kubernetes_events       Kubernetes Events
-  kafka                   Kafka consumer input plugin
-  fluentbit_metrics       Fluent Bit internal metrics
-  prometheus_scrape       Scrape metrics from Prometheus Endpoint
-  prometheus_textfile     Read Prometheus metrics from text files
-  tail                    Tail files
-  dummy                   Generate dummy data
-  head                    Head Input
-  health                  Check TCP server health
-  http                    HTTP
-  collectd                collectd input plugin
-  statsd                  StatsD input plugin
-  opentelemetry           OpenTelemetry
-  elasticsearch           HTTP Endpoints for Elasticsearch (Bulk API)
-  splunk                  Input plugin for Splunk HEC payloads
-  prometheus_remote_write Prometheus Remote Write input
-  event_type              Event tests for input plugins
-  nginx_metrics           Nginx status metrics
-  serial                  Serial input
-  stdin                   Standard Input
-  syslog                  Syslog
-  udp                     UDP
-  exec_wasi               Exec WASI Input
-  tcp                     TCP
-  mqtt                    MQTT, listen for Publish messages
-  forward                 Fluentd in-forward
-  random                  Random
-
-Processors
-  content_modifier        Modify the content of Logs, Metrics and Traces
-  labels                  Modifies metrics labels
-  metrics_selector        select metrics by specified name
-  opentelemetry_envelope  Package log records inside an OpenTelemetry Logs schema
-  sampling                Sampling
-  sql                     SQL processor
-
-Filters
-  alter_size              Alter incoming chunk size
-  aws                     Add AWS Metadata
-  checklist               Check records and flag them
-  ecs                     Add AWS ECS Metadata
-  record_modifier         modify record
-  sysinfo                 Filter for system info
-  throttle                Throttle messages using sliding window algorithm
-  type_converter          Data type converter
-  kubernetes              Filter to append Kubernetes metadata
-  modify                  modify records by applying rules
-  multiline               Concatenate multiline messages
-  nest                    nest events by specified field values
-  parser                  Parse events
-  expect                  Validate expected keys and values
-  grep                    grep events by specified field values
-  rewrite_tag             Rewrite records tags
-  log_to_metrics          generate log derived metrics
-  lua                     Lua Scripting Filter
-  stdout                  Filter events to STDOUT
-  geoip2                  add geoip information to records
-  nightfall               scans records for sensitive content
-  wasm                    WASM program filter
-
-Outputs
-  azure                   Send events to Azure HTTP Event Collector
-  azure_blob              Azure Blob Storage
-  azure_logs_ingestion    Send logs to Log Analytics with Log Ingestion API
-  azure_kusto             Send events to Kusto (Azure Data Explorer)
-  bigquery                Send events to BigQuery via streaming insert
-  counter                 Records counter
-  datadog                 Send events to DataDog HTTP Event Collector
-  es                      Elasticsearch
-  exit                    Exit after a number of flushes (test purposes)
-  file                    Generate log file
-  forward                 Forward (Fluentd protocol)
-  http                    HTTP Output
-  influxdb                InfluxDB Time Series
-  logdna                  LogDNA
-  loki                    Loki
-  kafka                   Kafka
-  kafka-rest              Kafka REST Proxy
-  nats                    NATS Server
-  nrlogs                  New Relic
-  null                    Throws away events
-  opensearch              OpenSearch
-  oracle_log_analytics    Oracle log analytics
-  plot                    Generate data file for GNU Plot
-  pgsql                   PostgreSQL
-  skywalking              Send logs into log collector on SkyWalking OAP
-  slack                   Send events to a Slack channel
-  splunk                  Send events to Splunk HTTP Event Collector
-  stackdriver             Send events to Google Stackdriver Logging
-  stdout                  Prints events to STDOUT
-  syslog                  Syslog
-  tcp                     TCP Output
-  udp                     UDP Output
-  td                      Treasure Data
-  flowcounter             FlowCounter
-  gelf                    GELF Output
-  websocket               Websocket
-  cloudwatch_logs         Send logs to Amazon CloudWatch
-  kinesis_firehose        Send logs to Amazon Kinesis Firehose
-  kinesis_streams         Send logs to Amazon Kinesis Streams
-  opentelemetry           OpenTelemetry
-  prometheus_exporter     Prometheus Exporter
-  prometheus_remote_write Prometheus remote write
-  s3                      Send to S3
-  vivo_exporter           Vivo Exporter
-  chronicle               Send logs to Google Chronicle as unstructured log
-```

administration/configuring-fluent-bit/classic-mode.md

@@ -1,4 +1,4 @@
-# Configure Fluent Bit using classic mode
+# Classic configuration files
 
 {% hint style="info" %}
 Fluent Bit classic mode configuration will be deprecated at the end of 2026.

administration/configuring-fluent-bit/security.md

@@ -1,4 +1,4 @@
-# YAML configuration
+# YAML configuration files
 
 <img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=864c6f0e-8977-4838-8772-84416943548e" />
 

administration/configuring-fluent-bit/unit-sizes.md

@@ -25,7 +25,7 @@ The plugin supports the following configuration parameters:
 
 | Key | Description | Default |
 | :--- | :--- | :--- |
-| `Buffer_Size` | Set the buffer size for HTTP client when reading responses from Kubernetes API server. The value must conform to the [unit size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. A value of `0` results in no limit, and the buffer will expand as-needed. If pod specifications exceed the buffer limit, the API response is discarded when retrieving metadata, and some Kubernetes metadata will fail to be injected to the logs. | `32k` |
+| `Buffer_Size` | Set the buffer size for HTTP client when reading responses from Kubernetes API server. The value must conform to the [unit size](../../administration/configuring-fluent-bit.md#unit-sizes) specification. A value of `0` results in no limit, and the buffer will expand as-needed. If pod specifications exceed the buffer limit, the API response is discarded when retrieving metadata, and some Kubernetes metadata will fail to be injected to the logs. | `32k` |
 | `Kube_URL` | API Server endpoint | `https://kubernetes.default.svc:443` |
 | `Kube_CA_File` | CA certificate file | `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt` |
 | `Kube_CA_Path` | Absolute path to scan for certificate files | _none_ |

administration/configuring-fluent-bit/yaml.md

@@ -22,8 +22,8 @@ The plugin supports the following configuration parameters:
 | `Event_Format` | Define event format to interact with Wasm programs: `msgpack` or `json`. Default: `json`. |
 | `Function_Name` | Wasm function name that will be triggered to do filtering. It's assumed that the function is built inside the Wasm program specified previously. |
 | `Accessible_Paths` | Specify the allowlist of paths to be able to access paths from Wasm programs. |
-| `Wasm_Heap_Size` | Size of the heap size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. |
-| `Wasm_Stack_Size` | Size of the stack size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. |
+| `Wasm_Heap_Size` | Size of the heap size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit.md#unit-sizes) for allowed values. |
+| `Wasm_Stack_Size` | Size of the stack size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit.md#unit-sizes) for allowed values. |
 
 ## Configuration example
 
@@ -46,7 +46,7 @@ pipeline:
       function_name: filter_function_name
       # Note: run Fluent Bit from the 'wasm_path' location.
       accessible_paths: /path/to/accessible
-    
+
   outputs:
     - name: stdout
       match: '*'
@@ -74,4 +74,4 @@ pipeline:
 ```
 
 {% endtab %}
-{% endtabs %}
+{% endtabs %}