Add shadow indexing

[vuldin]

This PR primarily enables shadow indexing configuration within the helm chart. But in the process the following changes are made:

• create shadow indexing example
• move/update load balancing example
• move/update security example
• update main README
• increment chart and redpanda image versions
• include all values.yaml properties with descriptions
• update PodDisruptionBudget API version
• make passing subdomain optional

This PR will move out of draft once the following tasks are complete:

• complete adding all properties and descriptions to values.yaml
• investigate possible issue with subdomain logic change in _helpers.tpl
• fix helm lint issues
• link issues this PR closes
• add test

I hope to have both of the above tasks done by EOD Monday.

closes #41 , closes #52

[vuldin]

I'm unable to add reviewers so I'll ping a few people who I've talked with about this PR here: @bpraseed @redp01 @larsenpanda @rkruze .

This PR looks big, but most of the updates relate to adding properties to the Values.yaml file. This was done so users of the chart will have a better idea of what is possible, and it may make it easier for this chart to be included in other upstream tools in the future.

I received some feedback between when I first posted this and now, and I've incorporated that feedback already.

[vuldin]

I didn't include some properties that exist. Here are some of them below. I didn't see many other examples where these properties were used, but they could be added if anyone thinks it would be valuable.

config:
  pandaproxy_client:
    sasl_mechanism:                                           # The SASL mechanism to use when connecting
    consumer_heartbeat_interval_ms: 500                       # Interval (in milliseconds) for consumer heartbeats
    consumer_rebalance_timeout_ms: 2000                       # Timeout (in milliseconds) for consumer rebalance
    consumer_request_max_bytes: 1048576                       # Max bytes to fetch per request
    consumer_request_timeout_ms: 100                          # Interval (in milliseconds) for consumer request timeout
    scram_username:                                           # Username to use for SCRAM authentication mechanisms
    produce_batch_delay_ms: 100                               # Delay (in milliseconds) to wait before sending batch
    produce_batch_size_bytes: 1048576                         # Number of bytes to batch before sending to broker
    consumer_session_timeout_ms: 300000                       # Timeout (in milliseconds) for consumer session
    produce_batch_record_count: 1000                          # Number of records to batch before sending to broker
    retry_base_backoff_ms: 100                                # Delay (in milliseconds) for initial retry backoff
    retries: 5                                                # Number of times to retry a request to a broker
    scram_password:                                           # Password to use for SCRAM authentication mechanisms
#    broker_tls: { enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 }  # TLS configuration for the brokers
    brokers: [{host: "172.17.0.7", port: 9092}]                 # List of address and port of the brokers

  schema_registry_client:
    sasl_mechanism:                                           # The SASL mechanism to use when connecting
    consumer_heartbeat_interval_ms: 500                       # Interval (in milliseconds) for consumer heartbeats
    consumer_rebalance_timeout_ms: 2000                       # Timeout (in milliseconds) for consumer rebalance
    consumer_request_max_bytes: 1048576                       # Max bytes to fetch per request
    consumer_request_timeout_ms: 100                          # Interval (in milliseconds) for consumer request timeout
    scram_username:                                           # Username to use for SCRAM authentication mechanisms
    produce_batch_delay_ms: 0                                 # Delay (in milliseconds) to wait before sending batch
    produce_batch_size_bytes: 0                               # Number of bytes to batch before sending to broker
    consumer_session_timeout_ms: 10000                        # Timeout (in milliseconds) for consumer session
    produce_batch_record_count: 0                             # Number of records to batch before sending to broker
    retry_base_backoff_ms: 100                                # Delay (in milliseconds) for initial retry backoff
    retries: 5                                                # Number of times to retry a request to a broker
    scram_password:                                           # Password to use for SCRAM authentication mechanisms
#    broker_tls: { enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 }  # TLS configuration for the brokers
    brokers: [{host: "172.17.0.7", port: 9092}]                 # List of address and port of the brokers

    # unknown
    enable_admin_api: true                                      # Enable admin API
    segment_bytes:                                          # for ballpark upper limit, divide disk size (in bytes) by number of partitions

  rpk:
    coredump_dir: /var/lib/redpanda/coredump
    enable_memory_locking: false                              # Enables memory locking.
    enable_usage_stats: false                                 # Send usage stats back to Redpanda
    overprovisioned: true                                     # This should be set to true unless CPU affinity is possible
    tune_aio_events: false                                    # Increases the number of allowed asynchronous IO events
    tune_ballast_file: false                                  # 
    tune_clocksource: false                                   # Syncs NTP
    tune_coredump: false                                      # Installs a custom script to process coredumps and save them to the given directory
    tune_disk_nomerges: false                                 # 
    tune_disk_scheduler: false                                # 
    tune_disk_write_cache: false                              # 
    tune_fstrim: false                                        # 
    tune_network: false                                       # 
    tune_swappiness: false                                    # 
    tune_transparent_hugepages: false                         # 

[vuldin]

This PR doesn't update to the latest Redpanda, instead it uses v21.11.16 due to the centralized config changes in v22.1. This issue needs to be resolved before updating to the latest Redpanda version.

[redp01]

@vuldin - thinking that it might be a good idea to add a basic helm test for the shadow indexing example. I think it should be fairly quick to add as it can just follow the existing tests.

[redp01]

@vuldin - thinking that it might be a good idea to add a basic helm test for the shadow indexing example. I think it should be fairly quick to add as it can just follow the existing tests.

[redp01]

Minor: Testing step by step according to the readme - I've noticed that the cert manager installation is using > to denote the next command line; however, its actually chained with the && operator so two of the > seem superfluous and prevent a straight copy paste of the command into bash.

[redp01]

@vuldin - I ran through the shadow indexing script in the readme from beginning to end. I took the Kind pathway on Mac OS 11.6.5. All seems to work perfectly and the watch on the minio bucket is a nice touch.

[vuldin]

Should this link be to another resource? I'm worried the SVG filename is generated by a release process (or something) and will no longer work at some point.

[vuldin]

Looks like I caused an issue with the Redpanda config when handling merge conflicts with previously merged commits to main. I'll take a look soon and force another push. I'm surprised by this because I was checking locally with ct lint --config redpanda/ci/ct.yaml. But it looks like this only checks lint and doesn't run tests.

[vuldin]

Switched this to draft. After discussing with @rkruze , I will split out the values.yaml modifications into a separate PR. This PR will be focused more on the shadow indexing changes, and will be again up for review once these changes are made.

[vuldin]

While creating this PR, it became evident there were many changes that were needed to this project before adding in a new feature. Most of these changes are found in #85 . Once PR 85 merged then a new (much smaller) shadow indexing PR will be created.

[theodrim]

Hello, is there any way to track progress on this - since #85 is merged for some time.

[vuldin]

hey @theodrim , thanks for mentioning this. I created a ticket to track: #142