Start expansion service containers before the controller container

[chamikaramj]

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[chamikaramj]

R: @ahmedabu98

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[ahmedabu98]

If we're starting both containers at the same time, we should probably have different port numbers so they don't conflict:

beam/sdks/java/transform-service/docker-compose/docker-compose.yml

Lines 24 to 35 in 0c55d38

	expansion-service-1:
	image: "apache/beam_java_expansion_service:${BEAM_VERSION}"
	restart: on-failure
	command: -id expansion-service-1 -port 5001
	volumes:
	- ${CREDENTIALS_VOLUME}:/credentials_volume
	environment:
	- GOOGLE_APPLICATION_CREDENTIALS=/credentials_volume/${GOOGLE_APPLICATION_CREDENTIALS_FILE_NAME}
	expansion-service-2:
	image: "apache/beam_python_expansion_service:${BEAM_VERSION}"
	restart: on-failure
	command: -id expansion-service-2 -port 5001

P.S. The transform service also has this same port number, should that be changed as well?

[ahmedabu98]

I'm not too familiar with how this works. Does this mean both expansion services will be built and run whenever we use the transform service?

[chamikaramj]

This is a docker compose feature. It allows specifying a dependency order between services (containers) of the Docker Compose service.

https://docs.docker.com/compose/compose-file/compose-file-v3/#depends_on

[ahmedabu98]

Ok I see. Maybe later we should consider specifying the services needed for transform-service. Most use-cases won't need both java and python expansion services. Not blocking though, this LGTM

[chamikaramj]

Most use-cases won't need both java and python expansion services

I'm hoping to use this as a generalized service that support all transforms. The main cost will be container download and storage. Once downloaded the expansion service containers can be started very quickly. Also, hoping to push changes to reduce size of the Java container in the future by removing the duplications due to using multiple shaded jars.

[chamikaramj]

"5001" is the internal port number of the containers (not the external port) so there's no port conflict here.

[chamikaramj]

Thanks.

[chamikaramj]

(test failures are unrelated)