Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an '--unload' option/subcommand to nycdb #97

Open
toolness opened this issue Apr 10, 2019 · 2 comments
Open

Add an '--unload' option/subcommand to nycdb #97

toolness opened this issue Apr 10, 2019 · 2 comments

Comments

@toolness
Copy link
Contributor

It would be really nice to have an --unload command to nycdb that simply undoes anything that --load does.

The simple version of this could just drop the tables mentioned in the YAML file, but a more awesome version would also be able to drop any derived tables (e.g. those hpd_registrations_grouped_by_bbl or whatever) and functions that were created... I have some code over in JustFixNYC/nycdb-k8s-loader that parses the actual SQL files to glean some of this information, if it'd be helpful, but maybe that's complicating things too much.

@aepyornis
Copy link
Collaborator

Sure, why not! Yup, it'd mostly be just dropping tables and functions.

By the way, you don't have to parse the sql files, verify.py has a list of tables that are created for each dataset (including the derived ones). I'll admit, though, that it's not the most obvious place for that constant, so perhaps we should put it somewhere else or include that information in the yaml files instead.

@toolness
Copy link
Contributor Author

Oooh yeah!! I really like the idea of putting verification-related metadata in the YAML file. Verification isn't documented in the ADDING_NEW_DATASETS.md guide and I have no idea how to do it... and to the extent that one of the best sources of ad-hoc documentation on our datasets so far is the YAML files, this would make it easier for folks to understand what derived tables are created for datasets!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants