BigQuery jobs.insert needs both upload (for configuration.load) and doit (for configuration.query, .extract and .copy)

[emk]

Hello! I've been experimenting with BigQuery and Rust and I've been I've been really impressed with the google-bigquery2 crate. I managed to get OAuth2 and some simple queries working, and everything worked beautifully.

However, I ran into an interesting complication. The hub.jobs().query(...) is meant for small queries that output a few hundred or a few thousand rows of JSON. Larger queries need to be run using jobs().insert(...) with a configuration.query structure, and the results need to be exported to Google Cloud Storage using the same API with a configuration.extract structure.

But this means that JobInsertCall actually needs both an upload method, and a doit method! Specifically, if you look at the documentation here, the breakdown appears to be:

• configuration.load: Needs upload.
• configuration.query: Needs a regular doit.
• configuration.extract: Needs a regular doit.
• configuration.copy: Needs a regular doit.

But JobInsertCall has a private doit method that only provides an implementation for upload using POST https://www.googleapis.com/upload/bigquery/v2/projects/projectId/jobs, and which has no access to regular doit using POST https://www.googleapis.com/bigquery/v2/projects/projectId/jobs.

The key code appears to be in mbuild.mako, which is set up to generate either a public doit function, or both a public upload function and a private doit function that can only be used to help implement upload.

The error that BigQuery gives me is:

Error: Bad Request (400): Uploads are only allowed with load jobs
    global: Uploads are only allowed with load jobs, invalid

I would be happy to submit a PR, but this involves a refactoring of mbuild.mako. Is this something that you would prefer to do yourself? Or would you be willing to explain how you'd like the code to be refactored?

I'm going to go ahead and modify my local copy to get rid of upload_action on this endpoint, regenerate everything, and verify that it works as expected. If it would be useful, I could also supply you with a test program.

Thank you for a very helpful and well-built library, and for any advice you can provide!

[emk]

Confirmed! If I remove the upload support from jobs.insert, then configuration.query works nicely! So the general strategy of offering both an upload method and a normal public doit method on JobInsertCall appears to be perfectly sound.

[emk]

Here's a version of google-bigquery2 which provides doit instead of upload: faradayio@7c314e5

As mentioned above, I've verified that configuration.query works as expected. So the proposed fix of generating both an upload and regular doit method version should work nicely.

[Byron]

Thanks so much for taking the time to write things down so that it's perfectly possible to follow you, and of course for the praise :).

I also think that the issue you encountered must be taken into consideration for the next iteration of this API generator here: #189 .

Your fix could be merged, but is likely to be overwritten next time someone runs make update-json. As of know, there is no mechanism for patching JSONS automatically, and possibly the generator should create 'working' versions from the originals anyway.
If you ever get an intuition how the current implementation could be fixed, please let me know or submit a PR - I am happy to merge and re-release.

[emk]

Oh, yeah, my "fix" isn't mergeable as is, because it breaks configuration.load while fixing configuration.query, configuration.extract and configuration.copy. That's just our internal workaround so I can test some other stuff.

The right fix, I think, it to change mbuild.mako to always generate a pub fn doit(&self), and to additionally generate a completely separate upload function when the JSON file indicates that is possible. This could do done with some copying and pasting in mkbuild.mako to (1) always make doit public with no stream or MIME argument, and to (2) fix upload to call a new upload_helper function instead of doit.

Does this make sense?

[Byron]

Yes it does, and the suggested fix should be relatively easy to do, too. These days I am rather afraid to touch the 'mako' based build system, merely because I haven't done so in a while and I know there are essentially no tests :D.

…

On Tue, Jun 5, 2018 at 2:40 PM Eric Kidd ***@***.***> wrote: Oh, yeah, my "fix" isn't mergeable as is, because it breaks configuration.load while fixing configuration.query, configuration.extract and configuration.copy. That's just our internal workaround so I can test some other stuff. The right fix, I think, it to change mbuild.mako to always generate a pub fn doit(&self), and to *additionally* generate a completely separate upload function when the JSON file indicates that is possible. This could do done with some copying and pasting in mkbuild.mako to (1) always make doit public with no stream or MIME argument, and to (2) fix upload to call a new upload_helper function instead of doit. Does this make sense? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#191 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD4hh21fp7X0q3OldTzMP45G4KuFmr8ks5t5nw9gaJpZM4Uamgo> .

[rossmeissl]

@Byron just curious if you plan to move forward on this?

[rossmeissl]

@Byron checking in on this

[rossmeissl]

@Byron any word?

[Byron]

Apologies for the late reply.
I have currently no plan of working on this project, but welcome any contributions.

[Byron]

I am closing this issue as since then a lot has changed. All crates are now async, available as v2.0 on crates.io and build against the latest API spec.
Please let me know in the comments if the issue is still relevant though.