log-processor: lambda failing because of missing variable 'batch_size'

[jkrivas]

Describe the bug

Using version 2.1.1 of solution for sending AWS WAF logs from S3 to OpenSearch.

When trying to upload a bigger batch of logs, it gets error 413 (too big batches sent). It then should automatically adjust batch size, however it fails with error that variable batch_size does not exist

Expected Behavior

For it to successfully send logs to OpenSearch.

Current Behavior

[INFO] PUT aws-waf/_bulk
[INFO] --> bulk_load response code 413
[ERROR] NameError: name 'batch_size' is not defined
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 44, in lambda_handler
    raise e
  File "/var/task/lambda_function.py", line 38, in lambda_handler
    getattr(sys.modules[__name__], func_name)(event)
  File "/var/task/lambda_function.py", line 84, in parse_sqs_event
    sqs.process_event(event)
  File "/var/task/event/event_parser.py", line 452, in process_event
    total, failed_records = self._bulk(records)
  File "/var/task/event/event_parser.py", line 124, in _bulk
    return idx_svc.bulk_load_idx_records(self._process_by_plugins(records))
  File "/var/task/idx/idx_svc.py", line 333, in bulk_load_idx_records
    self.adjust_bulk_batch_size()
  File "/var/task/idx/idx_svc.py", line 351, in adjust_bulk_batch_size
    if batch_size >= 4000:

Reproduction Steps

Try to load a big batch of logs to OpenSearch so that function adjust_bulk_batch_size gets invoked

Possible Solution

It seems it tries to reference global variable batch_size which is not defined. After manually defining the variable with

batch_size = int(os.environ.get("BULK_BATCH_SIZE", DEFAULT_BULK_BATCH_SIZE))

it started working.

Additional Information/Context

No response

Solution Version

2.1.1

AWS Region. e.g., us-east-1

No response

Other information

No response

[JoeShi]

will be fix in v2.2

[evalzy]

Fixed in V2.1.2