[BEAM-8108] Add Chicago Taxi Example running on Flink#10531
[BEAM-8108] Add Chicago Taxi Example running on Flink#10531Ardagan merged 4 commits intoapache:masterfrom
Conversation
|
@kkucharc Could you trigger some tests for me, please? After that: Thanks! |
|
Run Seed Job |
1 similar comment
|
Run Seed Job |
|
Run Chicago Taxi on Dataflow |
1 similar comment
|
Run Chicago Taxi on Dataflow |
|
Thanks for your help @aromanenko-dev! I'm investigating the failure |
|
@kamilwu Please, let me know if you will need any other help. |
|
Run Seed Job |
|
Run Chicago Taxi on Dataflow |
|
Run Chicago Taxi on Flink |
Without the fix, init action would fail if the `python` executable was Python 3.x
kamilwu
force-pushed
the
chicago-flink
branch
from
4944918 to
87f705c
Compare
|
retest this please |
|
Run Seed Job |
|
Run Chicago Taxi on Flink |
|
Run Chicago Taxi on Dataflow |
|
R: @pabloem Can I ask you for a review? |
| chicagoTaxiJob(delegate) | ||
| } | ||
|
|
||
| CronJobBuilder.cronJob( |
There was a problem hiding this comment.
PostcommitJobBuilder already adds periodic cron job
Any reason to add extra one?
There was a problem hiding this comment.
The cron job defined by PostcommitJobBuilder is executed every six hours, which is too frequent for our needs. The extra one overrides it with more desired setting (every 24 hours)
pabloem
left a comment
pabloem
left a comment
There was a problem hiding this comment.
I've reviewed almost everything (it LGTM btw) besides the file .test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Flink.groovy . If @Ardagan thinks it looks good, then this is fine to move forward.
If not, I'll take a look tomorrow morning!
Thanks.
pabloem
left a comment
pabloem
left a comment
There was a problem hiding this comment.
Just one comment about Python version. Other than that, LGTM.
| def numberOfWorkers = 5 | ||
|
|
||
| Docker publisher = new Docker(scope, LoadTestsBuilder.DOCKER_CONTAINER_REGISTRY) | ||
| publisher.publish(':sdks:python:container:py2:docker', 'python2.7_sdk') |
There was a problem hiding this comment.
I wonder if we should be using a py3 container straight out
There was a problem hiding this comment.
though maybe then we'd have to run everything in py3 (meaning the target :portable:py2:chicagoTaxiExample). LMK what you think.
There was a problem hiding this comment.
I ran the test in py3 (both SDK harness and the pipeline program) and there are some errors connected with dill package. What do you think about leaving it "as is" and creating a jira ticket to move chicago taxi to py3? We will have to do the migration anyway in the nearest future.
There was a problem hiding this comment.
|
Run Seed Job |
|
Run Chicago Taxi on Flink |
|
Thanks everyone! |
5 participants
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.