[BEAM-8273] Expand portability environment documentation

[ibzib]

Document the EXTERNAL environment.
Add additional instructions for using the PROCESS environment.

cc @functicons

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---		---	---
Java
Python		---			---	---
XLang	---	---	---		---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[tweise]

Thanks for taking this up! For external environment here are some notes in case you want to include:

External environment / worker pool can be used like this:

docker run --rm -p=50000:50000 gcr.io/apache-beam-testing/beam/sdks/snapshot/python3.6:20190904 --worker_pool=true

( or docker run --rm -p=50000:50000 localusername-docker-apache.bintray.io/beam/python:latest --worker_pool=true)

Add to pipeline options: --environment_type=EXTERNAL --environment_config=localhost:50000 

If you run the jobserver outside Docker, export BEAM_WORKER_POOL_IN_DOCKER_VM=1

[ibzib]

Thanks for the suggestions Thomas. It looks like BEAM_WORKER_POOL_IN_DOCKER_VM is only needed on macOS. Is that correct?

[tweise]

Thanks for the suggestions Thomas. It looks like BEAM_WORKER_POOL_IN_DOCKER_VM is only needed on macOS. Is that correct?

It is probably also needed for Windows.

[ibzib]

It is probably also needed for Windows.

Good point. I added your suggestions, PTAL.

[mxm]

Thanks for the suggestions Thomas. It looks like BEAM_WORKER_POOL_IN_DOCKER_VM is only needed on macOS. Is that correct?

It is probably also needed for Windows.

Originally, it was a MacOS specific workaround but it looks like it applies to Windows as well: https://docs.docker.com/docker-for-windows/networking/

[tweise]

@ibzib this entire section doesn't belong into the roadmap. I think this is a good opportunity to add a portability page to https://beam.apache.org/documentation/ and start adding info there.

[ibzib]

@ibzib this entire section doesn't belong into the roadmap. I think this is a good opportunity to add a portability page to https://beam.apache.org/documentation/ and start adding info there.

There's a page dedicated to "Runtime environments," but currently it only focuses on how to "customize, build, and push Beam SDK container images." I suppose that might be a logical place for this?

I should also probably update the pipeline options source with some of this information.

[tweise]

Yep, good find. "Runtime Environments" should probably be renamed "Containers" or something like that and then this could be added as "Portable Pipeline Environments"

[robertwb]

This isn't specific to Python.

[ibzib]

Removed the reference to Python. Also added a note that while these are Python options, they might apply to other SDKs as well. (Maybe a follow-up could be to fill this in for Java and Go.)

[robertwb]

There aren't any workers in this case. I would phrase this as "rather than starting up worker nodes, it calls back to the original process that submitted the job to process the data" or something like that.

[ibzib]

Removed the references to worker nodes.

[mxm]

@ibzib Do you want to merge/address the remaining suggestions?

[ibzib]

@tweise I moved this section to its own page.

[tweise]

Thanks for moving the page!

[tweise]

@ibzib please check if the JIRA should be closed: https://issues.apache.org/jira/browse/BEAM-8273