The Open-Source Analytics Development Platform

Metrics, Models, Jobs, Notebooks, End-to-end Column Lineage, and More!
Define your source of truth with an open standard
Slack · Website · Issues · Roadmap

Open Source Analytics Development Platform

demo.mp4

The problem: Today's data stack is highly fragmented, insecure, and expensive.

• Most modern data teams have at least 5-10 tools in their data stack. They often don't work well together.
• Today's closed ecosystem is vulnerable to security breaches and tool discontinuation.
• Vendor lock-in has led to exorbitant costs and rent-seeking behavior:

The Solution: Turntable, a post-modern data stack

• Open: Our whole repo is AGPL and always will be.
• Private: Self-deploy our open-source version or use our cloud product with our customer-deployed agent. Either way, your data never leaves your infrastructure.
• Integrated: With Turntable, all of your metrics, models, and data documents exist in the same context. We think this will allow for some totally new experiences. Think clicking on a dashboard and seeing exactly where a number came from.

✨ Features

• End-to-end Column lineage: See your data's journey from source to dashboard.
• Data catalog: See all your data assets in one place. We event write AI-generated descriptions for you (opt-in of course).
• Jobs (coming soon): Schedule and monitor your data pipelines.
• Pages (coming soon): Write and schedule analyses, reports, and data documents in an AI-powered, Notion-like interface.
• Full observability suite: Detect anomalies, monitor data quality, reduce cost, secure PII and more.
• Metrics: Define once, use anywhere, in pure SQL and GUI. No need to learn a new semantic layer framework.
• Subscriptions: Subscribe to data assets and get notified when they change. Each user gets their own dashboard
• Models: Allow team members to schedule and publish sql queries that depend on one another, with best practices already built in.
• CLI: Use the code editor you already love, but make secure onboarding a breeze.

🔌 Integrations

Data Sources: Postgres, Snowflake, BigQuery, DuckDB, Redshift, S3, and more.

BI Tools: Looker, Tableau, Metabase, PowerBI.

Data Transformation: dbt Core, dbt Cloud, SqlMesh

Saas (coming soon): Stripe, Zendesk, Hubspot, Salesforce, and more.

🔔 Stay up to date

Turntable launched its v0.1 on August 2024. Lots of new features are coming, and are generally released on a bi-weekly basis. Watch updates of this repository to be notified of future updates.

Check out our public roadmap

🙏 Credits

Our repo stands on the shoulders of some awesome open source projects. A big reason we decided to go open source is that there are lots of players in the data space who wrap open source projects in a proprietary layer and charge a lot for it. That didn't feel right to us.

With that context, we wanted to explicitly call out some of the projects we love and how we use them:

• Ag-Grid: Table component
• DataHub: metadata ingest engine
• Django: Core application framework
• Hatchet:*workflow scheduler
• Ibis: metrics layer and database connectors
• Instructor: structured AI output
• Lago: inspiration for our documentation
• Mintlify: docs site generator
• Nivo: charting
• Posthog: analytics
• Reactflow: lineage canvas
• Recharts: charting
• Rye: package management
• SQLGlot: sql/lineage parser
• TipTap: Text editor

🔖 License

Turntable is distributed under the AGPLv3 License.

💻 Deploy locally

Requirements

1. Install Docker on your machine;
2. Make sure Docker Compose is installed and available (it should be the case if you have chosen to install Docker via Docker Desktop); and
3. Make sure Git is installed on your machine.

Run the app

To start using Turntable

1. Clone the repository

# Get the code
git clone https://github.com/turntable-so/turntable.git

# Go to Turntable folder
cd turntable

2. Configure environment variables Create a .env.local file in the root of the project by running the following command

MacOS or Linux

bash generate_keys.sh

Windows

powershell -executionpolicy bypass -File .\generate_keys.ps1

No environment variables except the secrets generated by the commands above are required to run the app, but some functionality may be limited (e.g. AI-written documentation). See .env.example for a list of all available environment variables.

3. Start the app

To start the app you have two choices:

(A) Run Turntable with No demo resources

Run the following command:

docker compose --env-file .env.local up --build

Once the docker build is complete (a few minutes), you will see a line in the terminal like this: 'The app is ready! Visit http://localhost:3000/ to get started'. Once you do, open your browser and go to http://localhost:3000 to see the app running. Signup with a username and password to start using the app.

(B) Run Turntable with demo resources

If you'd like to also see the product with a demo postgres, dbt project, and metabase already connected, run:

docker compose --env-file .env.local -f docker-compose.demo.yml up --build

Once the docker build is complete (longer than above, usually 5+ minutes), you will see a line in the terminal like this: 'The app is ready! Visit http://localhost:3000/ to get started'. Once you do, open your browser and go to http://localhost:3000 to see the app running. The demo resources can be found in an account with user dev@turntable.so and password mypassword. Login with these credentials to see the demo resources, with associated lineage and asset viewer. If you'd like to start from a blank slate on this instance, simply sign up with a different email.

Analytics and tracking for the self-hosted version

Please note that Turntable, by default, tracks basic actions performed on your self-hosted instance, but you can easily opt out by setting the value of NEXT_PUBLIC_POSTHOG_KEY to "" in the docker-compose yml file you are using (e.g. docker-compose.yml or docker-compose.demo.yml). We do not track any telemetry in development (i.e. using docker-compose.dev.yml). For more information, please see our privacy policy.

☁️ Use our cloud-based product

Email us or visit our website to get started with our cloud product. Their our two vairants: a fully-hosted offering, and hybrid one, which includes a customer-deployed agent.

🚀 Getting the most out of Turntable

• See the documentation to learn more about all the features;
• Join our Slack community if you need help, or want to chat, we’re here to help;
• Follow us on Twitter or LinkedIn for the latest news;
• You can email us as well: team@turnable.so.

📝 Contributing

You can follow the instructions below to set up ourm development environment on your machine. This is intended for people interested in contributing to Turntable. If you just want to try Turntable on your local system, we recommend that you the instructions above to run a prod instance.

For now, we are not accepting pull requests from the community, but We are working on a process to make this possible in the future. For now, file an bug reports and feature requests as a GitHub issue, Roadmap submission or Slack post.

🧑‍💻 Development environment

To start the development environment, simply follow the instructions above to start the app, but change the final commmand to:

docker compose -f docker-compose.dev.yml --env-file .env.local up --build

Unlike the production environemnt, this supports hot reload. It also includes the demo resources described above.

Once everything starts (several minutes), run the command below to access a shell inside the container:

docker compose -f docker-compose.dev.yml exec worker bash

From here, backend tests can be run with pytest.