A decentralized compliance testing framework for Substrait implementations. Engine teams self-certify by running standardized test suites locally and generating structured compliance reports.
Highlights
- 8 Language SDKs: Java, Python, Rust, Go, C++, TypeScript, C#, and Scala
- 140 Function Test Files across 14 categories (arithmetic, string, comparison, datetime, geospatial, JSON, and more)
- TPC-H Benchmark — 22 industry-standard queries with binary and JSON Substrait plan formats
- TPC-DS Benchmark — 99 queries, 194 Substrait plans
- REST API — Spring Boot reference implementation for report submission and leaderboard (self-hosted)
- Interactive Demo — HTML dashboard with mock engines, query drill-down, and complexity filtering
- 10-Phase Framework — Validation, failure analysis, storage, analytics, and reproducibility
- CI/CD Workflows — 13 GitHub Actions workflows for SDK testing, API builds, and release
Getting Started
git clone https://github.com/IBM/substrait-compliance.git
cd substrait-compliance/demo
./runner/run-simple-demo.sh
See the README for full quick-start and SDK integration guides.
Known Issues
- Some SDKs have minor unused import warnings
- Performance benchmarking features are in beta
- REST API is a reference implementation; no hosted public service is provided at this time