Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[backport] [23.1.x] rm_stm: replace std::vector usage with fragmented_vector #9626

Merged
merged 4 commits into from
Mar 23, 2023

Conversation

bharathv
Copy link
Contributor

@bharathv bharathv commented Mar 22, 2023

A lot of state in rm_stm is backed by std::vector. With frequent snapshotting + non trivial pid counts large contiguous allocations cannot be satisfied if the memory is fragmented. This patch switches to using fragmented_vector for this usecase.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.1.x
  • v22.3.x
  • v22.2.x

Release Notes

Improvements

  • Improves memory management with transactions and idempotency by avoiding large contiguous allocations.

- Adds additional typedefs for vector compatibility.
- Implement front()
- Make push_back() accept a forwarding reference.
- Defines a custom move c-tor / assignment that guarantees empty()
after move similar to std::vector.

(cherry picked from commit c7dea50)
Uses the same underlying serialization as std::vector for compatibility.
Adds a bounds check to limit vector size to int32_t_max which was an
implicit assumption but a vector can be much bigger.

(cherry picked from commit 1a5dde7)
Addressing some comments post merge.

(cherry picked from commit a24e608)
@bharathv bharathv changed the title rm_stm: replace std::vector usage with fragmented_vector [backport] [23.1.x] rm_stm: replace std::vector usage with fragmented_vector Mar 22, 2023
@vshtokman vshtokman added this to the v23.1.3 milestone Mar 22, 2023
@bharathv
Copy link
Contributor Author

CI is green but doing another ci-repeat 3 as this can land in a release today.. just want to be sure there are no regressions.

@bharathv
Copy link
Contributor Author

/ci-repeat 3

@bharathv bharathv merged commit 02fa94e into redpanda-data:v23.1.x Mar 23, 2023
@bharathv bharathv deleted the rm_stm_frag_vector_231x branch March 23, 2023 15:53
@vshtokman vshtokman modified the milestones: v23.1.3, v23.1.4 Mar 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants