Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only use data that anyone can access in the dev workspace #802

Closed
archiewood opened this issue Apr 24, 2023 · 5 comments
Closed

Only use data that anyone can access in the dev workspace #802

archiewood opened this issue Apr 24, 2023 · 5 comments
Labels
dev-process issues related to developing evidence in this repo

Comments

@archiewood
Copy link
Member

Feature Description
In the development workspace today, we have a mixture of queries that reference different data sources. Some of this is by design (eg when testing different database types) but mostly this is incidental.
Importantly, developers who are not part of the core Evidence team, do not have access to the BiqQuery database.

Suggest short term:

  • Move all queries in spec to use local DuckDB (full support for types etc)

Longer term:

  • May make sense to not require queries to produce data, but data objects that are the same as the outputs from queries. Though I'd be slightly worried that this would be less good at catching bugs, for example if we change the data format.

Goal of Feature
All developers have parity in dev experience without needing access to Evidence data warehouse credentials.

@archiewood archiewood added the dev-process issues related to developing evidence in this repo label Apr 24, 2023
@archiewood archiewood changed the title Only use data that anyone can access in the development workspace Only use data that anyone can access in the dev workspace Apr 24, 2023
@hughess
Copy link
Member

hughess commented Apr 24, 2023

I think a pretty large % of the queries in there now are manually defined with unions, so just putting a duckdb file in the repo might get us most of the way there.

@archiewood
Copy link
Member Author

Agh got a really long way here and then realized that Vercel + DuckDB = Bad

Workaround here would be to switch to Netlify for dev workspace. Think this is probably the easiest path forwards.

@archiewood
Copy link
Member Author

Okay I've pinged it up on netlify and it works.

Proposal: we move evidence-development-workspace from Vercel -> Netlify, at least until we resolve that issue

@archiewood
Copy link
Member Author

Okay for now I have:

Things we could consider in the future:

  • A build:strict on the deploy preview, such that any failing queries throw.
  • This would require removing any components with error states from the deploy preview, so we'd need a different way to test these

@mcrascal
Copy link
Member

mcrascal commented May 5, 2023

Implemented in #809

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dev-process issues related to developing evidence in this repo
Projects
None yet
Development

No branches or pull requests

3 participants