aws_kinesis_streams partition_key_field broken in release 0.26.0+

[fleeco]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

While testing the aws_kinesis_streams sink we noticed odd behavior on the partition_key field for the records coming in.

It appeared they were getting random 16 byte entries no matter what we set the partition_key_field to.

On top of that, providing NO partition_key_field had the same effect - along with providing a partition key field that didn't exist (where the behavior is expected to not write to Kinesis at all)

I'm an absolute Rust noob, but started to try and debug what was going on. However once I reverted to an earlier version, the partition key works as expected.

The breaking change was introduced between 0.25.2 and 0.26.0. I'm going to try and take a stab at this over the weekend, but it could be a pretty serious breaking change for some people depending how they are using the partition key.

Thanks!

Configuration

No response

Version

0.27.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hi @fleeco !

Thanks for this detailed report!

Are you seeing error logs being output indicating that it couldn't find the partition key? Could you share the log output if so?

Sharing the full configuration would also be helpful to reproduce.

[fleeco]

heya!

So sadly - no log output at all that shows it's unable to find the key.

Here is the config

kinesis:
type: aws_kinesis_streams
inputs:
- router.kinesis
region: us-east-1
stream_name: xxxxxxxxxxxxx
partition_key_field: event_uuid
compression: none
encoding:
codec: json
buffer:
- type: memory
max_events: 50000
when_full: overflow
- type: disk
max_size: 17179869184
when_full: block

Running on vector 0.25.2 it works as expected (and if the key doesn't exist, it throws the message we'd expect)

Anything vector 0.26.0 + it just seems to ignore the partition_key_field completely no matter what you put in. And the output appears to be a random 16 bytes.

LMK if there is anything else I can do to help! Going to try and get my rust-learnin' on this weekend but we're just version locked at 0.25.2 till it's fixed :)

Thanks again!

[zamazan4ik]

@jszwedko @neuronull @fleeco I guess the root cause is here: https://github.com/vectordotdev/vector/blob/master/src/sinks/aws_kinesis/streams/config.rs#L147 After the refactoring the partition_key_field is mistakenly field with None in all cases.

I will send a fix.

[zamazan4ik]

@fleeco could you please check the Vector build from this branch (#16286) and test - does it resolve the issue for you or not? You will need to build Vector on your own (cargo build --release) and test it with your config. I have no ability to check the fix on my own but I am 99.99% percent sure the fix should work fine.

[fleeco]

Dude! You rock! This works perfectly :) Just tested it and it works like a charm.

[zpriddy]

@zamazan4ik - it seems like with the latest release 0.28.1 if you enable compression then the partition key also gets compressed.. resulting in the same garbage partition key and the events get thrown out..

[spencergilbert]

@zpriddy would you mind opening a new issue?

[zpriddy]

@zpriddy would you mind opening a new issue?

#16823