Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

prost-build: support boxing fields #802

Merged
merged 1 commit into from Apr 12, 2023
Merged

prost-build: support boxing fields #802

merged 1 commit into from Apr 12, 2023

Conversation

krallin
Copy link
Contributor

@krallin krallin commented Jan 25, 2023 •
edited

This allows the user to request boxing of specific fields. This is useful for enum types that might have variants of very different size.

If the large variants are infrequent, then that means we reduce the size of messages, which makes them use less memory, and requires less CPU time when moving them.

In Buck2, we encode our logs as protobuf, this commit shows the wins of using boxing:

facebook/buck2@afb3307

krallin added a commit to krallin/tonic that referenced this pull request Jan 25, 2023
@krallin
tonic-build: support boxing fields
e9fc54b 
This exposes tokio-rs/prost#802 in Tonic,
though obviously until that's merged and released this diff does
nothing.
krallin added a commit to krallin/tonic that referenced this pull request Jan 25, 2023
@krallin
tonic-build: support boxing fields
79d3315 
This exposes tokio-rs/prost#802 in Tonic,
though obviously until that's merged and released this diff does
nothing.
@krallin krallin mentioned this pull request Jan 25, 2023
Merged
krallin added a commit to krallin/tonic that referenced this pull request Jan 25, 2023
@krallin
tonic-build: support boxing fields
08f44c0 
This exposes tokio-rs/prost#802 in Tonic,
though obviously until that's merged and released this diff does
nothing.
krallin added a commit to krallin/tonic that referenced this pull request Jan 26, 2023
@krallin
tonic-build: support boxing fields
c67fc25 
This exposes tokio-rs/prost#802 in Tonic,
though obviously until that's merged and released this diff does
nothing.
LucioFranco
LucioFranco requested changes Feb 13, 2023
View reviewed changes
prost-build/src/lib.rs Show resolved Hide resolved
@krallin krallin force-pushed the boxed branch 2 times, most recently from 4fea4ad to 120b374 Compare March 31, 2023 16:46
@thoughtpolice thoughtpolice mentioned this pull request Apr 12, 2023
Closed
@krallin krallin force-pushed the boxed branch 2 times, most recently from 808dfd5 to b13750f Compare April 12, 2023 15:03
@krallin
prost-build: support boxing fields
8afb3e0 
This allows the user to request boxing of specific fields. This is useful for
enum types that might have variants of very different size.
LucioFranco
LucioFranco previously requested changes Apr 12, 2023
View reviewed changes
Copy link
Member

@LucioFranco LucioFranco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about the delay. Overall LGTM, I have one comment and I think it would be good maybe to add a section to the README w/ some info on boxing and how it can be useful. Feel free to point to the buck commit. Once, those two things are done then we can merge and I can get a point release out for yall.

prost-build/src/code_generator.rs
Comment on lines +313 to +321
&& ((type_ == Type::Message || type_ == Type::Group)
&& self
.message_graph
.is_nested(field.type_name(), fq_message_name))
|| (self
.config
.boxed
.get_first_field(&fq_message_name, field.name())
.is_some());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we move this to a fn since this is getting a bit complicated now and we reuse it below as well.. I think a fn + a doc comment to make it easier for future readers would go a long way here.

@LucioFranco LucioFranco dismissed their stale review April 12, 2023 17:34

@davidbarsky said he will do the follow up work.

@LucioFranco LucioFranco merged commit 6b5516a into tokio-rs:master Apr 12, 2023
11 checks passed
LucioFranco pushed a commit that referenced this pull request Apr 12, 2023
@krallin @LucioFranco
prost-build: support boxing fields (#802)
7b6742b 
This allows the user to request boxing of specific fields. This is useful for
enum types that might have variants of very different size.
@LucioFranco LucioFranco mentioned this pull request Apr 12, 2023
Merged
LucioFranco added a commit to hyperium/tonic that referenced this pull request Apr 14, 2023
@krallin @LucioFranco
tonic-build: support boxing fields (#1252)
65d909e 
This exposes tokio-rs/prost#802 in Tonic,
though obviously until that's merged and released this diff does
nothing.

Co-authored-by: Lucio Franco <luciofranco14@gmail.com>
Jasperav added a commit to Jasperav/prost that referenced this pull request Jun 8, 2023
@Jasperav @dfreese
@LucioFranco @damelLP @YoshikiTakashima @danielsn @Gilnaa @tottoto @oliverbrowneprima @neoeinstein @adpaco-aws @ldm0 @3541 @krallin @BKDaugherty
Syncnew (#2)
b41ff14 
* Allow file descriptor be generated without --include_source_info (tokio-rs#786)

* Allow file descriptor be generated without --include_source_info

The file descriptor sets generated by rules_proto in bazel don't have
this by default, so this allows some flexibility for users to reuse
results that are already available to them.  It's fairly trivial
adjustment so it seemed reasonable to me to allow the flexibility.

* revert breaking changes and add default

* release 0.11.5 (tokio-rs#788)

* Add message and enum attributes to prost-build (tokio-rs#784)

closes tokio-rs#783

* chore: Prepare 0.11.6 release (tokio-rs#794)

* chore: Added Kani to CI. (#1) (tokio-rs#798)

* Added Kani documentation. (tokio-rs#799)

* Fix issue with negative nanos in Duration::try_from(), and add tests (tokio-rs#803)

* Fix issue with negative nanos in Duration::try_from(), and add a unit test

* add proptest for negative nanos

* PR comment: remove spurious dbg

* Prevent spurious overflow in check_duration_roundtrip test (tokio-rs#804)

Co-authored-by: Lucio Franco <luciofranco14@gmail.com>

* Clarify `default_package_filename` documentation. (tokio-rs#809)

* Bump msrv to 1.60 (tokio-rs#814)

* chore(types): Remove including generated code (tokio-rs#801)

* chore: Update github action (tokio-rs#815)

Co-authored-by: Lucio Franco <luciofranco14@gmail.com>

* chore: Add cargo-machete to detect unused dependencies (tokio-rs#817)

* chore: Update msrv to 1.60 (tokio-rs#818)

* feat: Added try_normalize to Timestamp (tokio-rs#796)

Co-authored-by: Lucio Franco <luciofranco14@gmail.com>

* Update PropProof docs to note the need to submodule init (tokio-rs#805)

Co-authored-by: Lucio Franco <luciofranco14@gmail.com>

* feat(build): Add direct fds compile support (tokio-rs#819)

This commit adds two new compile functions that allow passing a
`FileDescriptorSet` and it will generate the Rust code. This allows
users to use libraries like `protox` directly and in an ergonomic way.

* release 0.11.7 (tokio-rs#821)

* fix: correct change in visibility of compiler module (tokio-rs#824)

Introduced in tokio-rs#801
Closes tokio-rs#823

* release 0.11.8 (tokio-rs#825)

* Add existing roundtrip test to Kani CI and avoid recursive submoduling (tokio-rs#828)

* Add existing roundtrip test to Kani CI

* Bump up Kani version

* Remove `v` from GA version

* Update to `syn@2` & `prettyplease@0.2` (tokio-rs#833)

* Fix corrupted tests and missing CI testing (tokio-rs#832)

Co-authored-by: Lucio Franco <luciofranco14@gmail.com>

* chore: Update to criterion 0.4 (tokio-rs#835)

* Fix build in directory not named `prost` (tokio-rs#839)

This library will fail to build with the following error when checked
out into a directory not named exactly `prost`:

  error: failed to load manifest for workspace member `/home/alex/src/prost-rs/tests/single-include`

  Caused by:
    failed to load manifest for dependency `prost`

  Caused by:
    failed to read `/home/alex/src/prost/Cargo.toml`

  Caused by:
    No such file or directory (os error 2)

This is because the `single-include` test depends on `prost` with the
path `../../../prost`. This patch fixes the issue by correcting the path
to `../..`.

* prost-build: support boxing fields (tokio-rs#802)

This allows the user to request boxing of specific fields. This is useful for
enum types that might have variants of very different size.

* chore: Update to baptiste0928/cargo-install@v2 (tokio-rs#840)

Co-authored-by: Lucio Franco <luciofranco14@gmail.com>

* Fix typo in bail message (tokio-rs#848)

---------

Co-authored-by: David Freese <freese@google.com>
Co-authored-by: Lucio Franco <luciofranco14@gmail.com>
Co-authored-by: damel_lp <dlambertpowell@gmail.com>
Co-authored-by: Yoshiki Takashima <ytakashi@andrew.cmu.edu>
Co-authored-by: Daniel Schwartz-Narbonne <danielsn@users.noreply.github.com>
Co-authored-by: Gilad Naaman <gilad@naaman.io>
Co-authored-by: tottoto <tottotodev@gmail.com>
Co-authored-by: Oliver Browne <109075559+oliverbrowneprima@users.noreply.github.com>
Co-authored-by: Marcus Griep <marcus@griep.us>
Co-authored-by: Adrian Palacios <73246657+adpaco-aws@users.noreply.github.com>
Co-authored-by: Donough Liu <liudingming@bytedance.com>
Co-authored-by: Alex O'Brien <3541@3541.website>
Co-authored-by: Thomas Orozco <thomas@orozco.fr>
Co-authored-by: Brendon Daugherty <brendon1097@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LucioFranco LucioFranco LucioFranco approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@krallin @LucioFranco