Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bigquery integration #280

Closed
AdeelK93 opened this issue Nov 14, 2022 · 4 comments · Fixed by #305
Closed

Bigquery integration #280

AdeelK93 opened this issue Nov 14, 2022 · 4 comments · Fixed by #305
Labels
proposal Proposal for a new feature in Goblet

Comments

@AdeelK93
Copy link
Contributor

I'd like to propose a new integration/decorator - bigquery. Remote functions are recently GA.

Basically, this would be similar to the app.post decorator, but also:

  • create a cloud_resource connection (if it doesn't exist), sharing the name of the goblet app
  • register the cloud function or cloud run as a remote function in bigquery
  • require the user to provide the name of the BQ remote function to create, as well as the function type signature

Interested in feedback on the idea!
@anovis anovis added the proposal Proposal for a new feature in Goblet label Nov 14, 2022
@anovis
Copy link
Collaborator

anovis commented Nov 14, 2022

I really like this idea! I didn't know about remote functions, but they are pretty cool and should be easy enough to integrate into the existing goblet framework as a new resource.

@anovis
Copy link
Collaborator

anovis commented Dec 11, 2022

@AdeelK93 we are working on adding this feature at the moment. Do you have a specific use case in mind already?

@AdeelK93
Copy link
Contributor Author

Awesome! My use-case: we store big data in BigQuery, but keep a lot of metadata in other systems, that are either only accessible via either API or a database flavor not yet supported by federated queries. This feature would allow me to correlate such data with metadata, joining SQL to the output of a Python function in a real-time manner.

Another related use-case would involve working with other real-time APIs, like economic data. Goblet let's us have reproducible and easy builds for such a pattern.

Let me know if there's anything else I can do to help with development!

@anovis anovis mentioned this issue Jan 16, 2023
Merged
@anovis anovis closed this as completed in 860062b Jan 17, 2023
@anovis
Copy link
Collaborator

anovis commented Jan 17, 2023

@AdeelK93 this feature is out in the latest release of 0.10.0 . Let us know how it goes and if any changes/ improvements are needed!

quajones pushed a commit to quajones/goblet that referenced this issue Oct 19, 2023
@diegoandresdiazespinoza @quajones
BQ Remote Function (goblet#305)
1e188a9 
Add BigQuery remote functions creation, test and docs. (closes goblet#280 )

```
    @app.bqremotefunction(dataset_id=...)
    def my_remote_function(x: str, y: str) -> str:
        return f"input parameters are {x} and {y}"
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
proposal Proposal for a new feature in Goblet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BQ Remote Function diegoandresdiazespinoza/goblet
2 participants
@anovis @AdeelK93