Skip to content

Remove DynamoDB and Bigtable storage backends - #536

Merged
thweetkomputer merged 2 commits into
mainfrom
agent/remove-dynamodb-bigtable-backends
Jul 29, 2026
Merged

Remove DynamoDB and Bigtable storage backends#536
thweetkomputer merged 2 commits into
mainfrom
agent/remove-dynamodb-bigtable-backends

Conversation

@thweetkomputer

@thweetkomputer thweetkomputer commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Context

DynamoDB and Bigtable are no longer supported storage backends, but their handlers, build-time options, dependency components, and documentation remained in the repository. Keeping these dead paths made the supported WITH_DATA_STORE matrix ambiguous and continued to build unused third-party SDK components.

Behavior before and after

Before, WITH_DATA_STORE advertised and accepted DYNAMODB and BIGTABLE, with legacy handler implementations and configuration flags still present.

After, the supported backends are embedded RocksDB plus the EloqDSS RocksDB, RocksDB Cloud, and EloqStore variants. Configuring either removed value fails explicitly with Unknown WITH_DATA_STORE.

Implementation

  • Remove the DynamoDB and Bigtable handlers, typed helpers, and scanners.
  • Remove their storage initialization, AWS SDK lifecycle conditionals, CMake definitions, and link dependencies.
  • Stop building the DynamoDB AWS SDK component and Bigtable google-cloud-cpp component while retaining the components used by S3/GCS backends.
  • Update architecture, build, dependency-manifest, and backend documentation.

Design decisions and alternatives

The backends are removed as a complete compatibility surface rather than hidden from the CMake cache only. This prevents stale macros or direct includes from preserving unsupported builds. AWS and Google Cloud dependencies remain because the supported S3 and GCS variants still require them.

Test plan

  • Unit/TCL tests
  • Integration or manual validation
  • Formatting/build checks
  • Compatibility or performance validation, when relevant

Commands and results:

cmake --build bld-rocksdb --parallel 16                    # passed
cmake --build bld-eloqstore-local --parallel 16            # passed
cmake -S . -B /tmp/eloqkv-invalid-dynamodb... -DWITH_DATA_STORE=DYNAMODB ...  # failed as expected
cmake -S . -B /tmp/eloqkv-invalid-bigtable... -DWITH_DATA_STORE=BIGTABLE ...  # failed as expected
bash -n .github/scripts/common.sh data_substrate/scripts/third_party/build-ubuntu2404.sh  # passed
git diff --check                                           # passed

clang-format-18 was unavailable in the workspace. The C/C++ changes are branch and file removals; no runtime/TCL tests were run.

Risk assessment

This intentionally breaks builds that still select DYNAMODB or BIGTABLE. Supported RocksDB and EloqStore variants were rebuilt successfully. Cloud dependency pruning is limited to the removed SDK components; S3/GCS components remain enabled.

Rollback plan

Revert this PR to restore the removed handlers, build options, and dependency components.

Reviewer guide

Start with CMakeLists.txt and core/src/storage_init.cpp, then review the deleted store_handler/{dynamo,bigtable}_* files and the dependency changes in scripts/third_party/build-ubuntu2404.sh.

Follow-up work

The EloqKV parent repository will carry this branch commit as its submodule pointer so parent CI can validate the combined change before this PR is merged.

Summary by CodeRabbit

  • New Features
    • Storage configuration now strictly supports RocksDB and EloqStore backends, including cloud-based RocksDB options.
  • Changes
    • Removed DynamoDB and Bigtable storage backend support; build and cloud integration options were narrowed accordingly.
  • Documentation
    • Updated architecture and fault-tolerance docs to reflect the supported backends.
    • Clarified table update/idempotence timestamp behavior in the storage API documentation.

@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 604a042d-e0ca-471d-99e4-5c4438c7a34a

📥 Commits

Reviewing files that changed from the base of the PR and between e907dca and 550fe3d.

📒 Files selected for processing (1)
  • tx_service/include/store/data_store_handler.h

Walkthrough

The change removes DynamoDB and BigTable support from build configuration and storage selection, restricts AWS setup to supported cloud paths, updates backend documentation, and revises UpsertTable rollback write_time documentation.

Changes

Backend consolidation

Layer / File(s) Summary
Narrow backend build selection
CMakeLists.txt, scripts/third_party/build-ubuntu2404.sh, third_party/manifest.yml, store_handler/eloq_data_store_service/CMakeLists.txt
Build options, library wiring, third-party components, and dependency consumers no longer select DynamoDB or BigTable.
Restrict runtime storage and AWS paths
core/include/data_substrate.h, core/src/data_substrate.cpp, core/src/storage_init.cpp
AWS setup and storage-handler selection are restricted to supported configurations.
Align backend documentation
CLAUDE.md, README.md, tx_service/README.md, docs/01-architecture-overview.md, docs/09-store-handler.md
Documentation now describes the remaining RocksDB and EloqStore backend paths.

Write-time documentation

Layer / File(s) Summary
Document rollback write_time calculation
tx_service/include/store/data_store_handler.h
The UpsertTable documentation specifies commit and recovery rollback write_time calculations.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

Suggested reviewers: liunyl

Poem

A rabbit hops through stores anew,
Rocks and EloqStore carry through.
Dynamo and BigTable fade,
AWS paths are neatly laid.
Docs now match the trail we made.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: removing DynamoDB and Bigtable storage backends.
Description check ✅ Passed The description follows the template well, covering context, behavior, implementation, design, tests, risk, rollback, reviewer guide, and follow-up work.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch agent/remove-dynamodb-bigtable-backends

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@thweetkomputer
thweetkomputer marked this pull request as ready for review July 28, 2026 00:50

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tx_service/include/store/data_store_handler.h`:
- Around line 132-135: The comment above the rollback timestamp behavior should
explicitly distinguish normal and recovery rollback rules: replace the undefined
“last_valid_ts...” wording with the normal base
max(kv_create_index_op_.write_time_, commit_ts_bound_) plus one, and document
that recovery rollback uses the timestamp base value. Preserve the existing
commit and write_time semantics while making the idempotence/overwrite invariant
clear.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 669a6370-4124-42d9-8cee-8ab88b768f22

📥 Commits

Reviewing files that changed from the base of the PR and between 82cea28 and e907dca.

📒 Files selected for processing (23)
  • CLAUDE.md
  • CMakeLists.txt
  • README.md
  • core/include/data_substrate.h
  • core/src/data_substrate.cpp
  • core/src/storage_init.cpp
  • docs/01-architecture-overview.md
  • docs/09-store-handler.md
  • scripts/third_party/build-ubuntu2404.sh
  • store_handler/bigtable_handler.cpp
  • store_handler/bigtable_handler.h
  • store_handler/bigtable_scanner.cpp
  • store_handler/bigtable_scanner.h
  • store_handler/dynamo_handler.cpp
  • store_handler/dynamo_handler.h
  • store_handler/dynamo_handler_typed.cpp
  • store_handler/dynamo_handler_typed.h
  • store_handler/dynamo_scanner.cpp
  • store_handler/dynamo_scanner.h
  • store_handler/eloq_data_store_service/CMakeLists.txt
  • third_party/manifest.yml
  • tx_service/README.md
  • tx_service/include/store/data_store_handler.h
💤 Files with no reviewable changes (9)
  • store_handler/dynamo_scanner.h
  • store_handler/bigtable_handler.h
  • store_handler/bigtable_scanner.h
  • store_handler/dynamo_handler_typed.cpp
  • store_handler/dynamo_handler.h
  • store_handler/dynamo_handler_typed.h
  • store_handler/dynamo_scanner.cpp
  • store_handler/bigtable_scanner.cpp
  • store_handler/bigtable_handler.cpp

Comment thread tx_service/include/store/data_store_handler.h Outdated
@thweetkomputer
thweetkomputer merged commit fda69ce into main Jul 29, 2026
10 checks passed
@thweetkomputer
thweetkomputer deleted the agent/remove-dynamodb-bigtable-backends branch July 29, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants