Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs site #40

Merged
merged 9 commits into from
May 31, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/workflows/publish-docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: Docs
on:
workflow_dispatch:

jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.x
- run: pip install mkdocs-material mike
- run: mkdocs gh-deploy --force
33 changes: 32 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,32 @@
# dbt-meshify
# dbt-meshify

dave-connors-3 marked this conversation as resolved.
Show resolved Hide resolved

[Click here for full package doucmentation](https://dbt-labs.github.io/dbt-meshify/)

## Overview

`dbt-meshify` is a dbt-core plugin that automates the management and creation of dbt-core model governance features introduced in dbt-core v1.5

These dbt-core features include:

1. __[Groups](https://docs.getdbt.com/docs/build/groups)__ - group your models into logical sets.
2. __[Contracts](https://docs.getdbt.com/docs/collaborate/govern/model-contracts)__ - add model contracts to your models to ensure consistent data shape.
3. __[Access](https://docs.getdbt.com/docs/collaborate/govern/model-access)__ - control the `access` level of models within groups
4. __[Versions](https://docs.getdbt.com/docs/collaborate/govern/model-versions)__ - create and increment versions of particular models.

## Installation

```bash
pip install dbt-meshify
```

## Basic Usage

```bash
dave-connors-3 marked this conversation as resolved.
Show resolved Hide resolved
# create a group of all models tagged with "finance"
dbt-meshify add-group finance --owner name Monopoly Man -s +tag:finance

# create a contract to a model
dbt-meshify add-contract --select my_public_model

```
1 change: 1 addition & 0 deletions conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
extensions = ['sphinx_click']
128 changes: 66 additions & 62 deletions dbt_meshify/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,12 @@
from .dbt_projects import DbtProject, DbtProjectHolder, DbtSubProject

# define common parameters
project_path = click.option("--project-path", type=click.Path(exists=True), default=".")
project_path = click.option(
"--project-path",
type=click.Path(exists=True),
default=".",
help="The path to the dbt project to operate on. Defaults to the current directory.",
)

exclude = click.option(
"--exclude",
Expand Down Expand Up @@ -36,67 +41,6 @@ def cli():
pass


@cli.command(name="connect")
@click.argument("projects-dir", type=click.Path(exists=True), default=".")
def connect(projects_dir):
"""
Connects multiple dbt projects together by adding all necessary dbt Mesh constructs

PROJECTS_DIR: The directory containing the dbt projects to connect. Defaults to the current directory.
"""
holder = DbtProjectHolder()

while True:
path_string = input("Enter the relative path to a dbt project (enter 'done' to finish): ")
if path_string == "done":
break

path = Path(path_string).expanduser().resolve()
project = DbtProject.from_directory(path)
holder.register_project(project)

print(holder.project_map())


@cli.command(name="split")
@exclude
@project_path
@select
@selector
def split():
"""
Splits dbt projects apart by adding all necessary dbt Mesh constructs based on the selection syntax.

Order of operations:
1. Regsiter the selected resources as a subproject of the main project
2. Add the resources to a group
2. Identifies the edges of the subproject with the remainder of the project
3. Adds contracts to all edges
"""
path_string = input("Enter the relative path to a dbt project you'd like to split: ")

holder = DbtProjectHolder()

path = Path(path_string).expanduser().resolve()
project = DbtProject.from_directory(path)
holder.register_project(project)

while True:
subproject_name = input("Enter the name for your subproject ('done' to finish): ")
if subproject_name == "done":
break
subproject_selector = input(
f"Enter the selector that represents the subproject {subproject_name}: "
)

subproject: DbtSubProject = project.split(
project_name=subproject_name, select=subproject_selector
)
holder.register_project(subproject)

print(holder.project_map())


@cli.command(name="add-contract")
@exclude
@project_path
Expand Down Expand Up @@ -154,3 +98,63 @@ def create_group(select, exclude, project_path, selector):
Add selected resources to a group
"""
pass


@cli.command(name="connect")
@click.argument("projects-dir", type=click.Path(exists=True), default=".")
def connect(projects_dir):
"""
!!! info
This command is not yet implemented

Connects multiple dbt projects together by adding all necessary dbt Mesh constructs
"""
holder = DbtProjectHolder()

while True:
path_string = input("Enter the relative path to a dbt project (enter 'done' to finish): ")
if path_string == "done":
break

path = Path(path_string).expanduser().resolve()
project = DbtProject.from_directory(path)
holder.register_project(project)

print(holder.project_map())


@cli.command(name="split")
@exclude
@project_path
@select
@selector
def split():
"""
!!! info
This command is not yet implemented

Splits dbt projects apart by adding all necessary dbt Mesh constructs based on the selection syntax.

"""
path_string = input("Enter the relative path to a dbt project you'd like to split: ")

holder = DbtProjectHolder()

path = Path(path_string).expanduser().resolve()
project = DbtProject.from_directory(path)
holder.register_project(project)

while True:
subproject_name = input("Enter the name for your subproject ('done' to finish): ")
if subproject_name == "done":
break
subproject_selector = input(
f"Enter the selector that represents the subproject {subproject_name}: "
)

subproject: DbtSubProject = project.split(
project_name=subproject_name, select=subproject_selector
)
holder.register_project(subproject)

print(holder.project_map())
7 changes: 7 additions & 0 deletions docs/assets/dbt-logo.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 6 additions & 0 deletions docs/commands.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# dbt-meshify Commands

::: mkdocs-click
:module: dbt_meshify.main
:command: cli
:prog_name: dbt-meshify
41 changes: 41 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# dbt_project_meshify
dave-connors-3 marked this conversation as resolved.
Show resolved Hide resolved

`dbt-meshify` is a dbt-core plugin that automates the management and creation of dbt-core model governance features introduced in dbt-core v1.5. Each command in the package will leverage your dbt project metadata to create and/or edit the files in your project to properly configure the models in your project with these governance features.

These dbt-core features include:

1. __[Groups](https://docs.getdbt.com/docs/build/groups)__ - group your models into logical sets.
2. __[Contracts](https://docs.getdbt.com/docs/collaborate/govern/model-contracts)__ - add model contracts to your models to ensure consistent data shape.
3. __[Access](https://docs.getdbt.com/docs/collaborate/govern/model-access)__ - control the `access` level of models within groups
4. __[Versions](https://docs.getdbt.com/docs/collaborate/govern/model-versions)__ - create and increment versions of particular models.

This package leverages the dbt-core Python API to allow users to use standard dbt selection syntax for each of the commands in this package (unless otherwise noted). See details on each of the specific commands available on the [commands page](commands.md)

## Basic Usage

Each of the available [commands page](commands.md) allows you to add one (or many) of the above features to a set of models specified by the selection syntax in the command.

The goal of this package is to make it more straightforward to apply to your project so that splitting apart a monolithic project into component projects is a more automated, dbt-tonic experience.

The process of splitting a dbt monolith apart roughly requires you to:

1. Determine what parts of your project should be grouped together into subprojects
2. Determine the access-level for the members of that group
3. Add model contracts to the elements that are public and accessed my members outside the group specified in (1)
4. (Optional) Add model versions to the public models to allow for development without impacting downstream stakeholders.

Here's how that might look for the process of creating a separate `finance` subproject in your dbt monolith.

```bash
# create a group of all models tagged with "finance"
# this command automatically detects models that should be public, and updates access levels
dbt-meshify add-group finance --owner name Monopoly Man -s +tag:finance

# create a contract to the publicly accessed nodes in that group
dbt-meshify add-contract --select group:finance,access:public

# optionally, add versions to your fct_payments model
dbt-meshify add-version --select fct_payments

```
Future releases of this package may also include features that allow users to fully split off groups of models into entirely new dbt projects.
72 changes: 72 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
site_name: dbt_meshify



theme:

palette:

# Palette toggle for light mode
- media: "(prefers-color-scheme: light)"
scheme: default
primary: custom
accent: custom
toggle:
icon: material/brightness-7
name: Switch to dark mode

# Palette toggle for dark mode
- media: "(prefers-color-scheme: dark)"
scheme: slate
primary: custom
accent: custom
toggle:
icon: material/brightness-4
name: Switch to light mode

# primary: black
logo: assets/dbt-logo.svg
name: material
features:
- navigation.footer
# - navigation.tabs
# - navigation.sections
- navigation.instant
- navigation.tracking
- content.action.edit
- toc.integrate # check feedback

extra_css:
- stylesheets/extra.css

extra:
version:
provider: mike

markdown_extensions:
- attr_list # needed to allow providing width
- md_in_html # to allow Markdown in details
- toc:
toc_depth: 3
permalink: "#"

# all for code blocks
- pymdownx.highlight:
anchor_linenums: true
line_spans: __span
pygments_lang_class: true
- pymdownx.inlinehilite
- pymdownx.snippets
- pymdownx.superfences
- mkdocs-click
- pymdownx.details # allow collapsible blocks
- admonition # allow call outs

repo_url: https://github.com/dbt-labs/dbt-meshify
repo_name: dbt-labs/dbt-meshify
edit_uri: edit/main/docs/

nav:
- Home: index.md
- Commands:
- List of commands: commands.md
Loading
Loading