[BUG] Application Web UI unavailable

[alexeyegorov]

Application Web UI unavailable

Expected behaviour

Run docker-compose up, run PySpark and see all the details of the running application through the web UI.

Current behaviour

After running docker-compose up and starting the pyspark session I am able to run everything, but I am not able to view the application web UI to see the stats of the finished jobs, DAGs, etc.

Steps to reproduce

1. Dockerfile with Spark 2.4.4
2. Run docker-compose up
3. Use Pyspark notebook to start a new session
4. Go to the Web UI localhost:4040 (while localhost:8080 is up).

Possible solutions (optional)

I have added port 4040 also to the Spark Master in the Dockerfile, but this didn't change anything.

Checklist

Please provide the following:

• Docker Engine version:

Client: Docker Engine - Community
 Cloud integration: 1.0.1
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 16:58:31 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:07:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

• Docker Compose version:

docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020

[alexeyegorov]

Adding 4040:4040 as port for jupyterlab service solves the issue. I don't quite understand why... does anybody else have this problem at all?

[rareal]

It looks like the Web UI is started by the SparkContext
https://spark.apache.org/docs/latest/monitoring.html
"Every SparkContext launches a Web UI, by default on port 4040, that displays useful information about the application"

As you start the SparkContext on the jupyterlab container, I guess that's the one that will have the server and hence needs the port binding.

[alexeyegorov]

@rareal Thanks for this hint! Logic is so helpful! :)
Nevertheless, I mean... probably as this is not a special case, but the ordinary usage of this docker image I think it might be helpful to add this single line to the docker-compose.yml file. I can do that and prepare a pull request. Does it make sence?

[rareal]

I think it makes sense, yes.
There's a Contributing ref too, I might contribute a windows build script later, if time permits :)

[alexeyegorov]

@rareal what is might be changed for Windows build script? For Mac OS I would just need to adjust the existing build script. Will you write a new one for the windows shell? That would be awesome, btw. Couple of colleagues are using windows and would really appreciate your work. :)

[rareal]

I'd write a powershell script, adapt the yml parsing (powershell also does not have a built in yml parser) and the rest seems pretty straightforward to adapt too

[andre-marcos-perez]

Hey @alexeyegorov and @rareal the solution proposed by #41 works like a charm. I'll be merging #41 to the develop branch for the next release! Thanks both for the contribution. @rareal I will accessing the #43 anytime soon.

[andre-marcos-perez]

@alexeyegorov and @rareal if you would like to be featured on the contributors, feel free to submit a pr on the README file, just make sure to follow the pattern.