[BEAM-4742] allow custom docker image in portable runner#5902
[BEAM-4742] allow custom docker image in portable runner#5902ryan-williams wants to merge 2 commits intoapache:masterfrom
Conversation
| output = counts | 'format' >> beam.Map(format_result) | ||
|
|
||
| out_dir = os.path.dirname(known_args.output) | ||
| if not FileSystems.exists(out_dir): |
There was a problem hiding this comment.
I believe the expectation should be that any output path should be created during pipeline execution and not by the driver program creating the pipeline.
Please revert this change to wordcount and fix the filesystem implementation to create any necessary directories instead.
There was a problem hiding this comment.
interesting, I originally made a change to LocalFileSystem to create directories on open, but I wasn't sure if that was the right semantics; it sounds like you're saying it is?
There was a problem hiding this comment.
I filed #5903 with that change; can close this in favor of that if that's what you prefer, thanks!
| @staticmethod | ||
| def default_docker_image(): | ||
| if 'USER' in os.environ: | ||
| if 'DOCKER_IMAGE' in os.environ: |
There was a problem hiding this comment.
This is already controlled by the flag --harness_docker_image:
Do you still want to make the default container selection be based off of DOCKER_IMAGE?
If yes, should it specify the full path and not assume the user wants the :latest suffix?
There was a problem hiding this comment.
ah, yea, I just saw the pipeline option for this as well! thanks for pointing it out.
There was a problem hiding this comment.
(I'll revert this part of the change, I don't think it's necessary)
2 participants
Allow specifying a docker image for the portable runner to use, via
DOCKER_IMAGEenv varAlso: make output directory in wordcount example, if it doesn't exist.
R: @angoenka
Post-Commit Tests Status (on master branch)