oscap-docker: remove atomic, fix environment variable configuration #1931

candrews · 2023-02-14T14:08:09Z

Remove usage of atomic from oscap-docker
The last release of atomic was 5 years ago.
The project is officially deprecated.
No linux distributions still package atomic.
Get the docker client api from the client
Getting the client api from the client reuses the client configuration.
This fixes an inconsistency where environment variable configuration is
used for the client but not for the client api.
This change therefore enables environment variable configuration (ex
DOCKER_HOST) to work.
Remove docker ping test
OscapDockerScan will throw an error if docker isn't running;
there's no need to create a separate docker client in a different way
just to do this ping test.

The last release of atomic was 5 years ago. The project is officially deprecated. No linux distributions still package atomic.

Getting the client api from the client reuses the client configuration. This fixes an inconsistency where environment variable configuration is used for the client but not for the client api. This change therefore enables environment variable configuration (ex DOCKER_HOST) to work.

OscapDockerScan will throw an error if docker isn't running; there's no need to create a separate docker client in a different way just to do this ping test.

Starting images to scan them has downsides, including that the image must be able to be started and stay running, which isn't always easy. For example, not all images contain `bash` or a shell at all. By instead using the approach taken by `oscap-podman` and extracting the image then scanning it (instead of starting the container and scanning the resulting mount), the container never needs to be run. This approach allows `oscap-docker` to scan any image (including those without `bash`).

candrews · 2023-02-19T02:43:05Z

@jan-cerny @evgenyz and anyone else,

Can you please take a look? oscap-docker is (more or less) unusable currently and this PR makes it work. I'd love to see this merged (or feedback provided so I can get it to that point).

The 4 failing checks are unrelated to this PR as far as I can tell.

Thank you in advance!

jan-cerny · 2023-02-20T10:57:51Z

@candrews Thank you very much for your PR! I agree that we should remove the dependency on Atomic, so I'm happy about your effort.

I was trying to test this PR. In RHEL and Fedora the Docker has been replaced by podman, and we're usually using oscap-podman. Therefore I tried the PR in an Ubuntu virtual machine. However, I have encountered a problem which I'm not sure if it's related to this PR or me using a wrong sequence of steps. I installed Ubuntu 22.04. I got the following:

user@ubuntu:~/openscap$ sudo docker images
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
ubuntu       22.04     58db3edaf2be   3 weeks ago   77.8MB
ubuntu       latest    58db3edaf2be   3 weeks ago   77.8MB
user@ubuntu:~/openscap$ sudo python3 utils/oscap-docker.in --oscap build/oscap_wrapper image 58db3edaf2be xccdf eval --profile ospp ~/ssg-ubuntu2204-ds.xml 
Creating a temporary container for the image...
Cannot create container for image ubuntu:22.04, ubuntu:latest.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/containers/create?name=tmp_oscap_4cf05444-b10c-11ed-9014-c731a7996a08

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/openscap/utils/oscap-docker.in", line 75, in <module>
    ODS = OscapDockerScan(args.scan_target, args.is_image, args.oscap_binary)
  File "/home/user/openscap/utils/oscap_docker_python/oscap_docker_util.py", line 73, in __init__
    raise e
  File "/home/user/openscap/utils/oscap_docker_python/oscap_docker_util.py", line 67, in __init__
    tmp_cont = self.client.create_container(
  File "/usr/lib/python3/dist-packages/docker/api/container.py", line 428, in create_container
    return self.create_container_from_config(config, name)
  File "/usr/lib/python3/dist-packages/docker/api/container.py", line 439, in create_container_from_config
    return self._result(res, True)
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/lib/python3/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.41/containers/create?name=tmp_oscap_4cf05444-b10c-11ed-9014-c731a7996a08: Bad Request ("invalid reference format")
!!! WARNING !!! This software has crashed, so you should check that no temporary container is still running
user@ubuntu:~/openscap$

Can you take a look? Have you encountered a similar problem? Are you on Ubuntu or on a different system? Which images and content do you recommend to use or are using personally? Thanks for any help.

candrews · 2023-02-21T20:41:21Z

Can you take a look? Have you encountered a similar problem?

I did some research, and "invalid reference format" usually means that the container name requested doesn't meet docker's container name requirements. It seems docker (but not podman!) limits container names to 30 characters, but this generated name is 47 characters long.

I've updated this PR with a commit to use the docker generated container id instead, eliminating the need to generate a name at all.

Can you please try your test again?

Are you on Ubuntu or on a different system?

I'm currently using Fedora 37. I've actually been testing this oscap-docker work against podman.

I need oscap-docker to work because I have systems that I need to support which use docker and do not have podman available.

Which images and content do you recommend to use or are using personally?

Here are some tests that I've performed successfully:

oscap-docker image registry.access.redhat.com/ubi8/ubi:8.1 xccdf eval --verbose ERROR --fetch-remote-resources --profile xccdf_org.ssgproject.content_profile_stig /usr/share/xml/scap/ssg/content/ssg-rhel8-ds.xml
oscap-docker image ubuntu:20.04 xccdf eval --verbose ERROR --fetch-remote-resources --profile xccdf_org.ssgproject.content_profile_stig /usr/share/xml/scap/ssg/content/ssg-ubuntu2004-ds.xml
oscap-docker image ubuntu:18.04 xccdf eval --fetch-remote-resources --profile MAC-3_Public --report report.html /home/candrews/Downloads/U_CAN_Ubuntu_18-04_V2R8_STIG_SCAP_1-2_Benchmark.xml

Thanks for any help.

You're welcome, I'm glad to have the opportunity work with you :-)

jan-cerny · 2023-02-22T13:44:27Z

I have discovered that the traceback that I posted above also happens with the current upstream maint-1.3 branch, that means the traceback isn't caused by the contents of this PR.

I also discovered that it doesn't happen when I try to scan a Fedora image on this Ubuntu host. Also, I was successful to scan the ubuntu:20.04 image on the same host. It seems that it has problems only with some images.

I've updated this PR with a commit to use the docker generated container id instead, eliminating the need to generate a name at all.
Can you please try your test again?

Great, but the same traceback still occurs. But as it isn't caused by the contents of this PR we can try to fix that later in a separate PR.

I'm currently using Fedora 37. I've actually been testing this oscap-docker work against podman.

Thanks for the hint. It's an interesting use-case, normally I would recommend using oscap-podman on the systems with podman.

So I tried this PR on a Fedora 37 machine as well. I have started the podman service which provides the API compatibility server and tried out the oscap-docker. And it seems to kind of work:

with the ubi:8.1 image, the scan was successful, and I got the results, but all rules are evaluated as notapplicable.
with the ubuntu:20.04 image it seems to work and some rules are evaluated and pass/fail results are given.
with the ubuntu:22.04 image, I got the same traceback that I got on Ubuntu machine in my first comment.

I need oscap-docker to work because I have systems that I need to support which use docker and do not have podman available.

I would recommend testing the changes also on these systems, or eg. on Ubuntu where the real docker is present, because the docker command on Fedora 37 is just an allias for podman. The podman CLI is compatible with the docker CLI but internally these 2 commands work differently.

So I think that this PR is actually OK, it doesn't introduce any new problems, but we should track the problems found (the traceback with ubuntu 20.04, the all rules notapplicable, etc.) and fix them later.

candrews · 2023-02-22T22:03:34Z

with the ubi:8.1 image, the scan was successful, and I got the results, but all rules are evaluated as notapplicable.

I ran oscap-podman registry.access.redhat.com/ubi8/ubi:8.1 xccdf eval --verbose ERROR --fetch-remote-resources --profile xccdf_org.ssgproject.content_profile_stig /usr/share/xml/scap/ssg/content/ssg-rhel8-ds.xml and it also reports everything as being notapplicable. So this appears to be an high level issue than docker vs podman

I would recommend testing the changes also on these systems, or eg. on Ubuntu where the real docker is present, because the docker command on Fedora 37 is just an allias for podman. The podman CLI is compatible with the docker CLI but internally these 2 commands work differently.

I'm aware and I'm also doing testing with real docker (and not just podman) as well.

I'm actually testing on Alpine too, and have been reporting and working with them to fix issues with its openscap packaging. I have been successfully testing this PR in Alpine on GitLab CI using a docker service.

candrews · 2023-02-24T22:06:29Z

What further can I do to facilitate the merging of this PR?

jan-cerny

I have reported the problems that have been discovered to upstream in #1942 and #1943.

I have checked that this PR doesn't add additional problems and that it really removes the Atomic dependency. I performed a scan of ubuntu:20.04 image successfully.

Great job!

candrews added 4 commits February 14, 2023 08:57

Remove usage of atomic from oscap-docker

5eabf7d

The last release of atomic was 5 years ago. The project is officially deprecated. No linux distributions still package atomic.

Remove docker ping test

aba83f3

OscapDockerScan will throw an error if docker isn't running; there's no need to create a separate docker client in a different way just to do this ping test.

jan-cerny self-assigned this Feb 20, 2023

jan-cerny added this to the 1.3.8 milestone Feb 20, 2023

oscap_docker_util.py: use container id instead of generating a name

497effd

This was referenced Feb 28, 2023

oscap-podman and oscap-docker evaluate all rules as notapplicable #1942

Closed

oscap-docker traceback when scanning ubuntu 22.04 image #1943

Open

jan-cerny approved these changes Feb 28, 2023

View reviewed changes

jan-cerny merged commit c8d3c87 into OpenSCAP:maint-1.3 Feb 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

oscap-docker: remove atomic, fix environment variable configuration #1931

oscap-docker: remove atomic, fix environment variable configuration #1931

Uh oh!

candrews commented Feb 14, 2023 •

edited

Loading

Uh oh!

candrews commented Feb 19, 2023

Uh oh!

jan-cerny commented Feb 20, 2023

Uh oh!

candrews commented Feb 21, 2023

Uh oh!

jan-cerny commented Feb 22, 2023

Uh oh!

candrews commented Feb 22, 2023

Uh oh!

candrews commented Feb 24, 2023

Uh oh!

jan-cerny left a comment

Uh oh!

Uh oh!

oscap-docker: remove atomic, fix environment variable configuration #1931

oscap-docker: remove atomic, fix environment variable configuration #1931

Uh oh!

Conversation

candrews commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

candrews commented Feb 19, 2023

Uh oh!

jan-cerny commented Feb 20, 2023

Uh oh!

candrews commented Feb 21, 2023

Uh oh!

jan-cerny commented Feb 22, 2023

Uh oh!

candrews commented Feb 22, 2023

Uh oh!

candrews commented Feb 24, 2023

Uh oh!

jan-cerny left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

candrews commented Feb 14, 2023 •

edited

Loading