Skip to content

New sgr cloud commands, fleshing out the splitgraph.yml (ex-repositories.yml), codegen for Splitgraph Cloud projects. #582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mildbyte merged 69 commits into from
Dec 16, 2021
Merged

New sgr cloud commands, fleshing out the splitgraph.yml (ex-repositories.yml), codegen for Splitgraph Cloud projects. #582

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
9566916
Move GQL queries into a separate module.
mildbyte Dec 6, 2021
ef6c8a9
Add `sgr cloud status` to get job ingestion status
mildbyte Dec 6, 2021
0f0b25b
Add `sgr cloud logs` to view ingestion job logs.
mildbyte Dec 6, 2021
2e08be6
Add `sgr cloud upload` for an end-to-end CSV upload.
mildbyte Dec 6, 2021
c4f8b9b
Pull `IngestionJobStatus` into a separate type.
mildbyte Dec 7, 2021
4d0c5e2
Add tests for `sgr cloud status`
mildbyte Dec 7, 2021
32b69bb
Add tests for `sgr cloud logs`.
mildbyte Dec 7, 2021
d1974b8
Factor table name deduplication out
mildbyte Dec 7, 2021
75f9967
Add tests for `sgr cloud upload`
mildbyte Dec 7, 2021
a5a5df6
Use unified GQL in `sgr cloud` instead of separate endpoints.
mildbyte Dec 7, 2021
0d552d4
Use ASCII spinners if SG_ASCII is set.
mildbyte Dec 7, 2021
628dded
Update docstrings and add new commands to the doc bundle.
mildbyte Dec 7, 2021
4f746a7
Add `sgr cloud sync` (trigger a one-off load)
mildbyte Dec 7, 2021
ebc6bc2
[WIP] Implement using `sgr cloud sync` with `repositories.yml`.
mildbyte Dec 7, 2021
434f3c2
Use `RepositoryType` in the CLI to parse repos.
mildbyte Dec 8, 2021
9f903e1
Add tests for `sgr cloud sync`.
mildbyte Dec 8, 2021
ad80d5d
Add `sgr cloud plugins` to list/search data plugins
mildbyte Dec 8, 2021
99744a0
Add `ruamel.yaml` to deps and regenerate the lockfile.
mildbyte Dec 8, 2021
3ec37ec
Fix linter errors
mildbyte Dec 8, 2021
0e1500e
Add `sgr cloud stub` (generate repositories.yml based on a plugin's J…
mildbyte Dec 8, 2021
4be2bf4
Delete `fetch_repositories_from_splitgraph` (superceded by `sgr cloud…
mildbyte Dec 9, 2021
b56c6e2
Use `ruamel.yaml` instead of PyYAML everywhere.
mildbyte Dec 9, 2021
533b1b5
Remove PyYAML, regenerate the lockfile.
mildbyte Dec 9, 2021
6db6ae9
Add tests for `sgr cloud plugins`
mildbyte Dec 9, 2021
20715ba
Add pytest-snapshot and use it to test `sgr cloud stub`.
mildbyte Dec 9, 2021
2005aa3
Replace more output checks with pytest-snapshot.
mildbyte Dec 9, 2021
a4bfcc4
Fix deprecation warning.
mildbyte Dec 9, 2021
76bfbe6
Use pytest-snapshot for `sgr cloud dump` tests
mildbyte Dec 9, 2021
217ada4
Use `default_flow_style=False` in YAML output
mildbyte Dec 9, 2021
372473a
Disable the spinner spam when stdout-forwarding.
mildbyte Dec 9, 2021
e0b40aa
Add `externalPlugin` and use it for `sgr cloud stub`
mildbyte Dec 9, 2021
38998e7
Move `raise_for_status` to the end of GQL error handling
mildbyte Dec 9, 2021
21fcecf
Add retry + backoff to GQL calls.
mildbyte Dec 9, 2021
477bb62
Add ability to merge repositories.yml files Compose-style
mildbyte Dec 13, 2021
1137a07
Add yml mixins to the CLI.
mildbyte Dec 13, 2021
1548501
Add GitHub Action file generation from a project seed.
mildbyte Dec 13, 2021
9bbfd74
Allow issuing anon requests to plugins GQL endpoints.
mildbyte Dec 13, 2021
d2218fe
Factor YML generation into a separate function.
mildbyte Dec 13, 2021
ef9447f
Add flag to skip setting externals in `sgr cloud load`
mildbyte Dec 13, 2021
a14d3ac
Fix default enum value (must be a string).
mildbyte Dec 13, 2021
f6cc712
Fix Click 8.x multiple=True errors with defaults
mildbyte Dec 13, 2021
ad36915
Fix linter issue
mildbyte Dec 13, 2021
d152a11
Generate sample dbt project from a list of repos
mildbyte Dec 13, 2021
fc1190d
Fix some visual bugs with stubbed YAMLs
mildbyte Dec 13, 2021
3147823
Allow repositories to be optional in repositories.yml
mildbyte Dec 13, 2021
8181c0a
Wire dbt project generation into `generate_project`.
mildbyte Dec 13, 2021
76c5f20
Remove spaces from separators to save a few bytes
mildbyte Dec 13, 2021
d3b68a5
Various type fixes
mildbyte Dec 13, 2021
74a94c1
Add `sgr cloud seed` (run `generate_project` from CLI)
mildbyte Dec 13, 2021
284aff5
Add `dbt_project.yml` to project generation tests.
mildbyte Dec 13, 2021
9395424
Make `test_generate_dbt_project` consistent (sort files)
mildbyte Dec 14, 2021
7c32b6a
Move splitgraph.yml-specific models into a separate module
mildbyte Dec 14, 2021
f68c96f
Rename `repositories.yml` to `splitgraph.yml`
mildbyte Dec 14, 2021
cfaebd7
Rename `RepositoriesYAML` -> `SplitgraphYAML`
mildbyte Dec 14, 2021
78e6a0f
Move project templates into a separate file.
mildbyte Dec 14, 2021
3122814
Add icons to builtin data sources.
mildbyte Dec 14, 2021
2479aa1
Remove `iconUrl` from the plugins query
mildbyte Dec 14, 2021
c1cf46d
Add logo for dbt. Make `get_icon` non-abstract.
mildbyte Dec 14, 2021
985ef90
Remove a straggler `icon_url`.
mildbyte Dec 14, 2021
747bd8f
Move non-CLI tests into `test/splitgraph/cloud`.
mildbyte Dec 14, 2021
6454a38
Allow setting initial repo visibility in `sgr cloud sync`
mildbyte Dec 15, 2021
3ff5481
Create private repositories in GitHub Actions.
mildbyte Dec 15, 2021
50085fc
Add `initial_private` to `sgr cloud load` as well.
mildbyte Dec 15, 2021
5076612
Set default deploy URL to `splitgraph.com`
mildbyte Dec 15, 2021
2403ada
Add a README to the autogenerated project.
mildbyte Dec 15, 2021
13cec46
Fix stylistic issues with the autogen project
mildbyte Dec 15, 2021
9306b8b
`sgr cloud load`: only set up credentials for repos that are being lo…
mildbyte Dec 15, 2021
953da08
Add new commands to the CLI reference.
mildbyte Dec 16, 2021
c308bf0
Log JSONSchema validation errors.
mildbyte Dec 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion docs/generate_reference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,14 @@
dump_c,
add_c,
token_c,
status_c,
logs_c,
upload_c,
seed_c,
validate_c,
stub_c,
plugins_c,
sync_c,
)
from splitgraph.commandline.engine import (
add_engine_c,
Expand Down Expand Up @@ -68,6 +76,7 @@
"cloud register",
"cloud login",
"cloud login-api",
"cloud token",
"cloud sql",
"cloud curl",
"cloud readme",
Expand All @@ -76,7 +85,14 @@
"cloud search",
"cloud dump",
"cloud load",
"cloud token",
"cloud sync",
"cloud status",
"cloud logs",
"cloud upload",
"cloud plugins",
"cloud stub",
"cloud validate",
"cloud seed",
],
),
]
Expand Down Expand Up @@ -108,6 +124,14 @@
"cloud dump": dump_c,
"cloud load": load_c,
"cloud token": token_c,
"cloud status": status_c,
"cloud logs": logs_c,
"cloud upload": upload_c,
"cloud sync": sync_c,
"cloud plugins": plugins_c,
"cloud stub": stub_c,
"cloud validate": validate_c,
"cloud seed": seed_c,
}


Expand Down
Loading