BEAM-8452 - TriggerLoadJobs.process in bigquery_file_loads schema is type str #10000
Conversation
| toxworkdir = {toxinidir}/target/{env:ENV_NAME:.tox} | ||
|
|
||
| [flake8] | ||
| ignore = E111,E114,E121,E125,E127,E129,E226,E302,E41,E502,W503,W504 |
There was a problem hiding this comment.
Why are these being ignored ? Please start a separate PR if you think we need this.
There was a problem hiding this comment.
Locally flake8 identified all of these as errors in the file I am trying to edit. Rather than refactor the file to match all of these rules, I thought it was simpler for myself and anyone else coming behind if flake8 knows that those rules aren't enforced. For example, a lot of the code was indented by multiples of 2 instead of 4, etc.
|
|
||
| if isinstance(schema, (str, unicode)): | ||
| schema = bigquery_tools.parse_table_schema_from_json(schema) | ||
| elif isinstance(schema, dict): |
There was a problem hiding this comment.
Could you explain the underlying issue here ? (couldn't find details in the JIRA).
Could the issue be that you are specifying an incompatible/incorrect schema object ? Can you try specifying schema as a TableSchema object ? For example, see here: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py#L80
If you are fixing an actual underlying issue here, please add a unit test.
There was a problem hiding this comment.
Apologies if the issue as described was not clear.
Basically - I was trying to creating a dataflow template, which requires using ValueProvider argument. I do not believe it is possible to pass a serialized object as the argument in this case.
I'm actually borrowing the logic from here: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L757
There was a problem hiding this comment.
Thanks. Can you please add a unit test.
There was a problem hiding this comment.
@chamikaramj I am working on it. Having a very difficult time trying to get the tests running locally.
There was a problem hiding this comment.
Thanks. This might help: https://cwiki.apache.org/confluence/display/BEAM/Python+Tips
|
cc: @pabloem |
|
🎆 🍾 🎉 🎈 Pull request #10000! |
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
3 participants
When trying to run a BigQueryFileLoads job, I receive the following error:
R: @chamikaramj
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.