feat: implement distributed query optimizations and advanced joins (Phase 5)

[poyrazK]

Description

This PR completes Phase 5 of the cloudSQL C++ migration and distributed optimization roadmap. It introduces key performance enhancements for cross-node queries, robust data redistribution infrastructure, and advanced join orchestration.

Key Changes

• Distributed Query Optimization:
  • Implemented Shard Pruning to intelligently route point queries to specific shards, reducing network traffic.
  • Added Aggregation Merging logic to the DistributedExecutor to coordinate global COUNT and SUM operations.
  • Introduced Broadcast Join orchestration, enabling efficient joins between small and large tables by redistributing data across the cluster.
• Execution Infrastructure:
  • Added BufferScanOperator to allow the Volcano engine to scan in-memory shuffle buffers.
  • Integrated ClusterManager buffering for staging redistributed data received via RPC.
• Networking & Serialization:
  • Implemented comprehensive Value and Tuple binary serialization in rpc_message.hpp.
  • Hardened the RPC layer with MSG_WAITALL and robust read patterns to prevent synchronization hangs.
• Parser Improvements:
  • Enhanced INSERT parsing for multi-row values.
  • Added support for COUNT(*) and improved function expression handling.
• Documentation:
  • Created a detailed technical record for all 5 phases in docs/phases/.
  • Updated README.md and architecture.md to reflect the new distributed capabilities.

Validation Results

• Core Tests: 28/28 passing.
• Distributed Tests: 6/6 passing (covering Shard Pruning, Shuffle, and Broadcast Join).
• Transaction Tests: 3/3 passing (2PC stability verified).
• Build: Clean build on AppleClang 17.0.0.

Steps to Test

1. Build the project: mkdir build && cd build && cmake .. && make -j
2. Run unit tests: ./sqlEngine_tests
3. Run distributed tests: ./distributed_tests
4. Run transaction tests: ./distributed_txn_tests

[coderabbitai]

Warning

Rate limit exceeded

@poyrazK has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 5 minutes and 34 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between e17c063 and 8c5a688.

📒 Files selected for processing (24)

• README.md
• docs/phases/PHASE_1_CORE.md
• docs/phases/PHASE_2_EXECUTION.md
• docs/phases/PHASE_3_SQL_CATALOG.md
• docs/phases/PHASE_4_CONSENSUS.md
• docs/phases/PHASE_5_OPTIMIZATION.md
• docs/phases/README.md
• include/common/cluster_manager.hpp
• include/distributed/distributed_executor.hpp
• include/distributed/shard_manager.hpp
• include/executor/operator.hpp
• include/executor/query_executor.hpp
• include/executor/types.hpp
• include/network/rpc_message.hpp
• plans/CPP_MIGRATION_PLAN.md
• plans/architecture.md
• src/distributed/distributed_executor.cpp
• src/executor/operator.cpp
• src/executor/query_executor.cpp
• src/main.cpp
• src/network/rpc_client.cpp
• src/network/rpc_server.cpp
• src/parser/parser.cpp
• tests/distributed_tests.cpp

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feature/phase-5-optimization

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}