Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(blooms): Add jsonnet lib for bloom filter components #12429

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

chaudum
Copy link
Contributor

@chaudum chaudum commented Apr 2, 2024

What this PR does / why we need it:

This PR adds the jsonnet lib for deploying bloom filter components (bloom-compactor for the write path, bloom-gateway for the read path) again.

Special notes for your reviewer:

This lib has been originally added with #11397, #11530, #11741 and removed with #11984.


⚠️ 🚧 ⚠️ Bloom filters are still an experimental feature. Do not enable the flag $._config.use_bloom_filters unless you know what you're doing and/or actively developing the feature.


Example configuration:

{
  _config+: {

    bloom_compactor+: {
      enabled: true,
      replicas: 3,
      pvc_size: '64Gi',
      pvc_class: 'fast-dont-retain',
    },

    bloom_gateway+: {
      enabled: true,
      replicas: 3,
      use_local_ssd: true,
      hostpaths: [
        '/mnt/disks/ssd%d' % x
        for x in std.range(0, 3)
      ],
      node_selector: {
        'has-local-ssd': 'true',
      },
      tolerations: [
        {
          key: 'type',
          operator: 'Equal',
          value: 'local-ssd',
          effect: 'NoSchedule',
        },
      ],
    },

    loki+:: {
      storage_config+: {
        bloom_shipper+: {
          download_parallelism: 4,
          max_query_page_size: '32MiB',
          metas_cache+: {
            default_validity: '48h',
          },
          blocks_cache: {
            ttl: '48h',
          },
        },
      },
      bloom_compactor: {
        enabled: true,
        compaction_interval: '6h',
      },
      bloom_gateway: {
        enabled: true,
        block_query_concurrency: 4,
        worker_concurrency: 4,
      },
    },

  },

  bloom_gateway_args+:: {
    'bloom.blocks-cache.soft-limit': '1200GiB',
    'bloom.blocks-cache.hard-limit': '1500GiB',
    'bloom.shipper.working-directory': std.join(',', ['/data%d/blooms/blocks' % x for x in std.range(0, 3)]),
  },
}

@chaudum chaudum force-pushed the chaudum/bloomfilter-jsonnet branch 2 times, most recently from 7ba8cb6 to a8cfe46 Compare April 2, 2024 14:46
@chaudum chaudum marked this pull request as ready for review April 2, 2024 14:46
@chaudum chaudum requested a review from a team as a code owner April 2, 2024 14:46
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
@chaudum chaudum force-pushed the chaudum/bloomfilter-jsonnet branch from a8cfe46 to 01adaef Compare April 16, 2024 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant