Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incompatible_proto_output_v2: new protobuf structure for aquery output #10358

Open
joeleba opened this issue Dec 3, 2019 · 0 comments
Open

incompatible_proto_output_v2: new protobuf structure for aquery output #10358

joeleba opened this issue Dec 3, 2019 · 0 comments

Comments

@joeleba
Copy link
Member

@joeleba joeleba commented Dec 3, 2019

Flag: --incompatible_proto_output_v2
Available since: 2.0
Will be flipped in: TBD

Motivation

Reducing the output size of aquery and avoiding OOMs.

Changes

1. Change ids field in aquery's proto output from string to uint32.
While this is the recommended best practice for proto, it makes the output unnecessarily big. Changing the type to uint32 should cover our use cases, while decreasing the output size.

2. Streaming aquery proto output.
Currently aquery's proto output is generated in a monolithic manner, i.e. the whole ActionGraphContainer is constructed in memory and then at the end printed to an output file. This requires the whole proto object to live in memory, which might cause OOMs. By restructuring the proto format and stream the result instead, we can avoid this situation.

Migration

TBD

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.