Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement new server storage #1016

Merged
merged 13 commits into from
Jul 1, 2024
Merged

Implement new server storage #1016

merged 13 commits into from
Jul 1, 2024

Conversation

spetz
Copy link
Collaborator

@spetz spetz commented Jun 25, 2024

This is a new, custom implementation of the server storage based on a "command-sourcing" log, which is required for the future replication purposes. The existing streams/topic/partitions/segments storage remains unchanged. This implementation removes the usage of Sled embedded DB and comes with a built-in migration.

@spetz spetz added new functionality New functionality config Configuration (client side or server side) change tui Command line interface (iggy-cmd/iggy-cli) related change server iggy-servere related change sdk Change related to sdk (client) API rust Pull requests that update Rust code api API interface related changes labels Jun 25, 2024
@spetz spetz force-pushed the new-storage branch 2 times, most recently from 6486fb7 to 3dcdf2a Compare June 25, 2024 19:44
BREAKING_CHANGES for API and SDK but comes with a built-in data migration for existing Sled DB
integration/tests/server/scenarios/system_scenario.rs Outdated Show resolved Hide resolved
integration/tests/server/scenarios/user_scenario.rs Outdated Show resolved Hide resolved
integration/tests/state/mod.rs Show resolved Hide resolved
server/src/compat/storage_conversion/converter.rs Outdated Show resolved Hide resolved
server/src/state/file.rs Outdated Show resolved Hide resolved
server/src/state/mod.rs Outdated Show resolved Hide resolved
server/src/state/system.rs Outdated Show resolved Hide resolved
server/src/state/system.rs Show resolved Hide resolved
server/src/streaming/systems/system.rs Show resolved Hide resolved
integration/tests/streaming/segment.rs Outdated Show resolved Hide resolved
sdk/src/cli/topics/get_topics.rs Show resolved Hide resolved
sdk/src/http/users.rs Show resolved Hide resolved
sdk/src/utils/expiry.rs Show resolved Hide resolved
server/src/binary/mapper.rs Show resolved Hide resolved
server/src/configs/validators.rs Outdated Show resolved Hide resolved
server/src/versioning.rs Show resolved Hide resolved
hubcio
hubcio previously approved these changes Jun 28, 2024
numinnex
numinnex previously approved these changes Jun 28, 2024
@spetz spetz dismissed stale reviews from numinnex and hubcio via a685ca2 June 29, 2024 17:05
@coveralls
Copy link
Collaborator

coveralls commented Jul 1, 2024

Pull Request Test Coverage Report for Build 9741717848

Details

  • 2901 of 4466 (64.96%) changed or added relevant lines in 184 files are covered.
  • 64 unchanged lines in 18 files lost coverage.
  • Overall coverage decreased (-2.9%) to 81.045%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sdk/src/binary/mapper.rs 19 20 95.0%
sdk/src/cli/personal_access_tokens/get_personal_access_tokens.rs 3 4 75.0%
sdk/src/cli/users/get_users.rs 1 2 50.0%
sdk/src/personal_access_tokens/create_personal_access_token.rs 17 18 94.44%
sdk/src/utils/duration.rs 4 5 80.0%
server/src/binary/handlers/consumer_groups/create_consumer_group_handler.rs 21 22 95.45%
server/src/channels/commands/clean_messages.rs 1 2 50.0%
server/src/compat/message_conversion/snapshots/message_snapshot.rs 0 1 0.0%
server/src/configs/displays.rs 1 2 50.0%
server/src/http/consumer_groups.rs 28 29 96.55%
Files with Coverage Reduction New Missed Lines %
server/src/streaming/streams/storage.rs 1 92.31%
server/src/binary/handlers/consumer_groups/delete_consumer_group_handler.rs 1 96.15%
server/src/streaming/systems/system.rs 1 74.13%
sdk/src/command.rs 1 0.0%
server/src/http/jwt/storage.rs 1 71.64%
server/src/http/users.rs 1 94.0%
server/src/streaming/topics/consumer_groups.rs 1 88.52%
server/src/streaming/systems/users.rs 1 89.23%
server/src/configs/system.rs 1 86.09%
sdk/src/models/permissions.rs 2 86.33%
Totals Coverage Status
Change from base Build 9741578452: -2.9%
Covered Lines: 23559
Relevant Lines: 29069

💛 - Coveralls

numinnex
numinnex previously approved these changes Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@iggy-rs iggy-rs deleted a comment from coveralls Jul 1, 2024
@coveralls
Copy link
Collaborator

coveralls commented Jul 1, 2024

Pull Request Test Coverage Report for Build 9747859729

Details

  • 2901 of 4483 (64.71%) changed or added relevant lines in 184 files are covered.
  • 65 unchanged lines in 19 files lost coverage.
  • Overall coverage decreased (-3.0%) to 80.994%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sdk/src/binary/mapper.rs 19 20 95.0%
sdk/src/cli/personal_access_tokens/get_personal_access_tokens.rs 3 4 75.0%
sdk/src/cli/users/get_users.rs 1 2 50.0%
sdk/src/personal_access_tokens/create_personal_access_token.rs 17 18 94.44%
sdk/src/utils/duration.rs 4 5 80.0%
server/src/binary/handlers/consumer_groups/create_consumer_group_handler.rs 21 22 95.45%
server/src/channels/commands/clean_messages.rs 1 2 50.0%
server/src/compat/message_conversion/snapshots/message_snapshot.rs 0 1 0.0%
server/src/configs/displays.rs 1 2 50.0%
server/src/http/consumer_groups.rs 28 29 96.55%
Files with Coverage Reduction New Missed Lines %
server/src/streaming/streams/storage.rs 1 92.31%
server/src/binary/handlers/consumer_groups/delete_consumer_group_handler.rs 1 96.15%
server/src/streaming/systems/system.rs 1 74.13%
sdk/src/command.rs 1 0.0%
server/src/http/jwt/storage.rs 1 71.64%
server/src/http/users.rs 1 94.0%
cli/src/credentials.rs 1 90.23%
server/src/streaming/topics/consumer_groups.rs 1 88.52%
server/src/streaming/systems/users.rs 1 89.23%
server/src/configs/system.rs 1 86.09%
Totals Coverage Status
Change from base Build 9741578452: -3.0%
Covered Lines: 23558
Relevant Lines: 29086

💛 - Coveralls

@spetz spetz merged commit faf7f9f into master Jul 1, 2024
17 checks passed
@spetz spetz deleted the new-storage branch July 1, 2024 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api API interface related changes config Configuration (client side or server side) change new functionality New functionality rust Pull requests that update Rust code sdk Change related to sdk (client) API server iggy-servere related change tui Command line interface (iggy-cmd/iggy-cli) related change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants