Skip to content

service: Fix tag deserialization for large messages#552

Merged
bkeryan merged 3 commits intomainfrom
users/bkeryan/fix-field-id
Dec 13, 2023
Merged

service: Fix tag deserialization for large messages#552
bkeryan merged 3 commits intomainfrom
users/bkeryan/fix-field-id

Conversation

@bkeryan
Copy link
Copy Markdown
Collaborator

@bkeryan bkeryan commented Dec 12, 2023

What does this Pull Request accomplish?

Tags are i32 varints, but we were only deserializing the first byte.

Simplify some related details:

  • Decoding a varint returns the updated position, so we don't need to use encoder._TagSize()
  • We can directly construct a memory view once instead of constructing a BytesIO for each loop iteration.

Note

This reuses an internal+underscored protobuf function, google.protobuf.internal.decoder._DecodeSignedVarint32. This code was already using a different internal+underscored protobuf function, google.protobuf.internal.encoder._TagSize.

For xydata support, we forked the unsigned _DecodeVarint, but I didn't do that here.

In the long run, I think we should generate descriptors so that we can stop using protobuf internals and let protobuf do the heavy lifting: #35

Why should this Pull Request be merged?

Fixes AB#2602618: Protobuf Error on Measurement run [MeasurementLink UI Editor/InstrumentStudio]

What testing has been done?

Ran poetry run pytest -v.
Manually tested the service attached to the bug report with my users/bkeryan/fix-field-id-1.2 branch (targeting releases/1.2).

Tags are i32 varints, but we were only deserializing the first byte.

Simplify some related details:
- Decoding a varint returns the updated position, so we don't need to use encoder._TagSize()
- We can directly construct a memory view once instead of constructing a BytesIO for each loop iteration.
Copy link
Copy Markdown
Contributor

@pbirkhol-ni pbirkhol-ni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't very familiar with this code so I took some time to familiarize myself. What you did seems reasonable.

Comment thread tests/unit/test_serializer.py Outdated
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Dec 13, 2023

Test Results

       30 files  ±  0         30 suites  ±0   37m 36s ⏱️ - 1m 59s
     546 tests +  2       546 ✔️ +  2         0 💤 ±0  0 ±0 
13 360 runs  +60  12 300 ✔️ +60  1 060 💤 ±0  0 ±0 

Results for commit d06fee9. ± Comparison against base commit 0d19a55.

♻️ This comment has been updated with latest results.

@bkeryan bkeryan merged commit 011a22b into main Dec 13, 2023
@bkeryan bkeryan deleted the users/bkeryan/fix-field-id branch December 13, 2023 15:34
bkeryan added a commit that referenced this pull request Dec 13, 2023
* tests: Add test cases for big messages (100 fields)

* service: Fix tag deserialization for large messages

Tags are i32 varints, but we were only deserializing the first byte.

Simplify some related details:
- Decoding a varint returns the updated position, so we don't need to use encoder._TagSize()
- We can directly construct a memory view once instead of constructing a BytesIO for each loop iteration.

* tests: Use pytest.approx instead of choosing exactly comparable floats
bkeryan added a commit that referenced this pull request Dec 13, 2023
)

service: Fix tag deserialization for large messages (#552)

* tests: Add test cases for big messages (100 fields)

* service: Fix tag deserialization for large messages

Tags are i32 varints, but we were only deserializing the first byte.

Simplify some related details:
- Decoding a varint returns the updated position, so we don't need to use encoder._TagSize()
- We can directly construct a memory view once instead of constructing a BytesIO for each loop iteration.

* tests: Use pytest.approx instead of choosing exactly comparable floats
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants