HEALTHCHECK Instructions?

[rwmajor2]

I have a general question. We base some of our Docker images on the Docker-stacks Dockerfiles. We are following CISP Docker Security hardening guidelines and one of the checklist items is:

 Ensure that HEALTHCHECK instructions have been added to container images
	Guidance:  Docker engine periodically checks the running container instances against that instruction to ensure containers are still operational

Can anyone provide any suggestions on what this is and how we may implement it in docker-stacks? According to Docker help documentation, a HEALTHCHECK might be something like:

HEALTHCHECK --interval=5m --timeout=3s CMD curl -f http://localhost/ || exit 1

Thoughts?

@GrahamDumpleton

[GrahamDumpleton]

Can't see what health checks have got to do with security hardening. They are an operations feature to ensure your application is running. Unless they think that your application not running may have been caused by hackers and so qualifies somehow as a security event. Also be aware that defining health checks in a container image itself only really pertains to Docker's container run time. They aren't used by other container platforms such as Kubernetes. In Kubernetes health checks are defined in the separate deployment configuration of Kubernetes where they more rightly belong. So not sure if it really belongs in the container image itself. It should be really part of how you deploy things in the container platform.

[rwmajor2]

Thanks @GrahamDumpleton, that's fair enough. Thanks for the feedback.

FYI, below is the "rationale" from the CISP guideline:

An important security control is that of availability. Adding the HEALTHCHECK instruction to your container image ensures that the Docker engine periodically checks the running container instances against that instruction to ensure that containers are still operational.
Based on the results of the health check, the Docker engine could terminate containers which are not responding correctly, and instantiate new ones.

[romainx]

Hello @rwmajor2

It's a late answer however I hope it could be useful to someone.

Standard HTTP probe

As far as I know there is currently no specific "health" end point available on Jupyter, it seems to be confirmed by this issue jupyter/notebook#1857.
Since notebook are protected it's not possible to use a standard HTTP health check like.

# I'm using wget since curl is not available
HEALTHCHECK CMD wget -q --spider http://127.0.0.1:8888 > /dev/null || exit 1

This will always return HTTP 405 and the wget command the 8 exit status, meaning
"Server issued an error response".

$ wget -q --spider http://127.0.0.1:8888/
# [W 12:07:40.913 NotebookApp] 405 HEAD / (127.0.0.1) 0.72ms referer=None

$ echo $?
# 8

An alternative

As an alternative it's possible to check if the Jupyter process is running through pgrep.

HEALTHCHECK CMD pgrep "jupyter" > /dev/null || exit 1

Here is the result

$ docker ps             
# CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                             PORTS                    NAMES
# 162adb12d75d        jupyter/base-notebook   "tini -g -- start-no…"   29 seconds ago      Up 28 seconds (health: starting)   0.0.0.0:8888->8888/tcp   vibrant_agnesi

$ docker ps
# CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                    PORTS                    NAMES
# 162adb12d75d        jupyter/base-notebook   "tini -g -- start-no…"   34 seconds ago      Up 33 seconds (healthy)   0.0.0.0:8888->8888/tcp   vibrant_agnesi

You can change the HEALTHCHECK settings (interval, timeout, etc.) as explained in the documentation.

Hope it helps. Please tell us if this is the case.
Best

[lmeyerov]

We used to do pgrep "jupyter" as described above, but find Jupyter kernels are prone to getting wedged without crashing, e.g., 100% CPU or IO, so we find this check not so great for availability in practice. Don't have an alt, however.

[mathbunnyru]

I think this is actually possible in a reliable way!

jupyter/notebook#1857 (comment)

It should be something like this:
HEALTHCHECK CMD curl --fail http://localhost:8888/api || exit 1

I checked that /api returns 200 for lab, notebook, nbclassic, server and retro jupyter commands.

Also, I'm not sure we should add this to our docker files at this point - people might using custom command, which do not actually launch jupyter subcommand.
So, I think, I will add this to docs and it should be fine.

[lmeyerov]

Ah, so curl -f (so responsive to HTTP error codes), nice, thanks!

[mathbunnyru]

Ah, so curl -f (so responsive to HTTP error codes), nice, thanks!

Thanks, fixed 👍

[carlosefr]

This change doesn't seem to work when the image is used in Jupyter Hub. The URL used in the health check fails with a 404.