Unable to run zaproxy on mac m1 using docker and podman both.

[anarang]

Hi,

I got a Mac M1 recently and I am trying to run the zaproxy tests against a config by placing the open api url and target url as expected in the config/config.yml and then following steps below but getting the below error.

Any help would be appreciated, thanks.

rapidast (master %) $ docker-compose up zaproxy
[+] Running 1/0
 ⠿ Container zaproxy  Created                                                                                                                                                                                                                                              0.1s
Attaching to zaproxy

Trying to run the scan as per the README file:

$ docker-compose exec zaproxy python /zap/scripts/apis_scan.py reports                                                                                                                                        1
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/usr/local/lib/python3.8/dist-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x4002245400>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=8090): Max retries exceeded with url: http://zap/JSON/core/action/newSession/?apikey=cnmeemn7jp7ijd8rl5u14q40v8&name=%2Fzap%2Fresults%2Freports%2Fsessions%2F20220419-170600%2Fsession1&overwrite=True (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4002245400>: Failed to establish a new connection: [Errno 111] Connection refused')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/zap/scripts/apis_scan.py", line 313, in <module>
    create_session(session_fullpath_name)
  File "/zap/scripts/apis_scan.py", line 17, in create_session
    zap.core.new_session(name=session_name, overwrite=True)
  File "/usr/local/lib/python3.8/dist-packages/zapv2/core.py", line 357, in new_session
    return six.next(six.itervalues(self.zap._request(self.zap.base + 'core/action/newSession/', params)))
  File "/usr/local/lib/python3.8/dist-packages/zapv2/__init__.py", line 200, in _request
    data = self._request_api(url, get)
  File "/usr/local/lib/python3.8/dist-packages/zapv2/__init__.py", line 180, in _request_api
    response = self.session.get(url, params=query, proxies=self.__proxies, verify=False)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 513, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPConnectionPool(host='127.0.0.1', port=8090): Max retries exceeded with url: http://zap/JSON/core/action/newSession/?apikey=cnmeemn7jp7ijd8rl5u14q40v8&name=%2Fzap%2Fresults%2Freports%2Fsessions%2F20220419-170600%2Fsession1&overwrite=True (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4002245400>: Failed to establish a new connection: [Errno 111] Connection refused')))

[anarang]

I tried to use podman as well but no luck there as well.
Here are the steps followed:

 $ sh runenv.sh                                                                                   
['podman', '--version', '']
using podman version: 4.0.3
** excluding:  set()
podman stop -t 10 zaproxy
Error: no container with name or ID "zaproxy" found: no such container
exit code: 125
podman rm zaproxy
Error: no container with name or ID "zaproxy" found: no such container
exit code: 1
['podman', '--version', '']
using podman version: 4.0.3
** excluding:  set()
podman stop -t 10 zaproxy
Error: no container with name or ID "zaproxy" found: no such container
exit code: 125
podman rm zaproxy
Error: no container with name or ID "zaproxy" found: no such container
exit code: 1
['podman', '--version', '']
using podman version: 4.0.3
** excluding:  set()
['podman', 'network', 'exists', 'rapidast_default']
podman create --name=zaproxy --label io.podman.compose.config-hash=123 --label io.podman.compose.project=rapidast --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=rapidast --label com.docker.compose.project.working_dir=/Users/anarang/work/security_testing/rapidast --label com.docker.compose.project.config_files=podman-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=zaproxy -v /Users/anarang/work/security_testing/rapidast/scripts:/zap/scripts:Z -v /Users/anarang/work/security_testing/rapidast/config:/zap/config:Z -v /Users/anarang/work/security_testing/rapidast/results:/zap/results:Z -v /Users/anarang/work/security_testing/rapidast/policies:/zap/policies:Z --net rapidast_default --network-alias zaproxy -p 8091:8090 -u zap -i --tty docker.io/owasp/zap2docker-stable /bin/bash -c pip install jinja2 && zap.sh -cmd -addonupdate && zap.sh -daemon -port 8090 -config api.key=cnmeemn7jp7ijd8rl5u14q40v8 -config database.newsession=3 -config database.newsessionprompt=false -addoninstall ascanrulesBeta
Error: statfs /Users/anarang/work/security_testing/rapidast/policies: no such file or directory
exit code: 125
podman start -a zaproxy
Error: no container with name or ID "zaproxy" found: no such container
exit code: 125

Attempted to fix it by mounting the volumes during podman init but it did not work as expected:

$ podman machine init -v /Users/anarang/work/security_testing/rapidast/scripts:/zap/scripts -v /Users/anarang/work/security_testing/rapidast/config:/zap/config --volume /Users/anarang/work/security_testing/rapidast/results:/zap/results -v /Users/anarang/work/security_testing/rapidast/policies:/zap/policies 
Extracting compressed file
Image resized.
Machine init complete
To start your machine run:
 
	podman machine start
 
anarang@anarang-mac:~/work/security_testing/rapidast (master %) $ podman machine start
Starting machine "podman-machine-default"
INFO[0000] waiting for clients...                       
INFO[0000] new connection from  to /var/folders/hy/xkhlnqsx3nq50twn8z7j_1280000gn/T/podman/qemu_podman-machine-default.sock 
Waiting for VM ...
INFO[0029] Socket forward established: /Users/anarang/.local/share/containers/podman/machine/podman-machine-default/podman.sock to /run/user/501/podman/podman.sock 
Mounting volume... /Users/anarang/work/security_testing/rapidast/scripts:/zap/scripts
Mounting volume... /Users/anarang/work/security_testing/rapidast/config:/zap/config
Mounting volume... /Users/anarang/work/security_testing/rapidast/results:/zap/results
Mounting volume... /Users/anarang/work/security_testing/rapidast/policies:/zap/policies
 
This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command: 
 
	podman machine set --rootful
 
API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.
 
Machine "podman-machine-default" started successfully
anarang@anarang-mac:~/work/security_testing/rapidast (master %) $ sh runenv.sh       
['podman', '--version', '']
using podman version: 4.0.3
** excluding:  set()
podman stop -t 10 zaproxy
Error: no container with name or ID "zaproxy" found: no such container
exit code: 125
podman rm zaproxy
Error: no container with name or ID "zaproxy" found: no such container
exit code: 1
['podman', '--version', '']
using podman version: 4.0.3
** excluding:  set()
podman stop -t 10 zaproxy
Error: no container with name or ID "zaproxy" found: no such container
exit code: 125
podman rm zaproxy
Error: no container with name or ID "zaproxy" found: no such container
exit code: 1
['podman', '--version', '']
using podman version: 4.0.3
** excluding:  set()
['podman', 'network', 'exists', 'rapidast_default']
['podman', 'network', 'create', '--label', 'io.podman.compose.project=rapidast', '--label', 'com.docker.compose.project=rapidast', 'rapidast_default']
['podman', 'network', 'exists', 'rapidast_default']
podman create --name=zaproxy --label io.podman.compose.config-hash=123 --label io.podman.compose.project=rapidast --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=rapidast --label com.docker.compose.project.working_dir=/Users/anarang/work/security_testing/rapidast --label com.docker.compose.project.config_files=podman-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=zaproxy -v /Users/anarang/work/security_testing/rapidast/scripts:/zap/scripts:Z -v /Users/anarang/work/security_testing/rapidast/config:/zap/config:Z -v /Users/anarang/work/security_testing/rapidast/results:/zap/results:Z -v /Users/anarang/work/security_testing/rapidast/policies:/zap/policies:Z --net rapidast_default --network-alias zaproxy -p 8091:8090 -u zap -i --tty docker.io/owasp/zap2docker-stable /bin/bash -c pip install jinja2 && zap.sh -cmd -addonupdate && zap.sh -daemon -port 8090 -config api.key=cnmeemn7jp7ijd8rl5u14q40v8 -config database.newsession=3 -config database.newsessionprompt=false -addoninstall ascanrulesBeta
Trying to pull docker.io/owasp/zap2docker-stable:latest...
Getting image source signatures
Copying blob sha256:ab1803ca01d13278d3f24cd6b01cfc64bfa11d925cd439ccdc105569cc1beac0
Copying blob sha256:4d32b49e2995210e8937f0898327f196d3fcc52486f0be920e8b2d65f150a7ab
Copying blob sha256:508c6e8a66835e1ed685d0a2b1daf454c387a5f44a556b7e92c6b49ec764f534
Copying blob sha256:40a9f14ce1d95c7859e78166945482cbd33b0230866686de04e7b293ad5cb0cf
Copying blob sha256:75bf84448705188a6b21cc1ee25d7de5f17bec3ed62a54b8e5e08e535ba46a71
Copying blob sha256:9e3e68bd3b71ebc9524d353764bc55a5d46c4eba490aafd622c4f3509c03011b
Copying blob sha256:75bf84448705188a6b21cc1ee25d7de5f17bec3ed62a54b8e5e08e535ba46a71
Copying blob sha256:508c6e8a66835e1ed685d0a2b1daf454c387a5f44a556b7e92c6b49ec764f534
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:40a9f14ce1d95c7859e78166945482cbd33b0230866686de04e7b293ad5cb0cf
Copying blob sha256:ab1803ca01d13278d3f24cd6b01cfc64bfa11d925cd439ccdc105569cc1beac0
Copying blob sha256:45ac76f9466f59d32c306c30631c907796761702ab36cd8c0075882e2a922cc2
Copying blob sha256:9e3e68bd3b71ebc9524d353764bc55a5d46c4eba490aafd622c4f3509c03011b
Copying blob sha256:90c756377282d6149c9ae3c3b0259031258355afaa854536391fa3a2a83051a9
Copying blob sha256:4d32b49e2995210e8937f0898327f196d3fcc52486f0be920e8b2d65f150a7ab
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:b143399d2488e9c6f1d8be1d86dbf45dc09bc008ba013ba436429f84f79c4be7
Copying blob sha256:90c756377282d6149c9ae3c3b0259031258355afaa854536391fa3a2a83051a9
Copying blob sha256:b143399d2488e9c6f1d8be1d86dbf45dc09bc008ba013ba436429f84f79c4be7
Copying blob sha256:c40c28208a72d2c4a752991323922cd939831bd7e34a41570acf8020e27108c7
Copying blob sha256:45ac76f9466f59d32c306c30631c907796761702ab36cd8c0075882e2a922cc2
Copying blob sha256:65ea597f12b42aac543091c45a17fd16d30094b1f6410595162e10379ab02a80
Copying blob sha256:65ea597f12b42aac543091c45a17fd16d30094b1f6410595162e10379ab02a80
Copying blob sha256:1d8bb9a4c673524e9ad767d0afe86a56101174c827b98ba44f88efd8ccbf4c9f
Copying blob sha256:c40c28208a72d2c4a752991323922cd939831bd7e34a41570acf8020e27108c7
Copying blob sha256:d0e66d3dab1d90e396577f3c657be29a7f7a3bf0e8dd37b362d6989f44e50bd5
Copying blob sha256:d0e66d3dab1d90e396577f3c657be29a7f7a3bf0e8dd37b362d6989f44e50bd5
Copying blob sha256:c5fa38ddc0da499e62849dea126b587ad026e496fccbf84dcad3ed723438b573
Copying blob sha256:1d8bb9a4c673524e9ad767d0afe86a56101174c827b98ba44f88efd8ccbf4c9f
Copying blob sha256:009adc6ba4ea2786547fca8973107533ef6617ef6eb7d78385bbbf65737aeb50
Copying blob sha256:c5fa38ddc0da499e62849dea126b587ad026e496fccbf84dcad3ed723438b573
Copying blob sha256:978b9e9305429b27a63a1e040a0924161a78d0d1109352a5173ccee50a9a077e
Copying blob sha256:009adc6ba4ea2786547fca8973107533ef6617ef6eb7d78385bbbf65737aeb50
Copying blob sha256:108d11132fe129ce1f2876c57b5642bd34e6e24166a525517702f146bd2a2610
Copying blob sha256:108d11132fe129ce1f2876c57b5642bd34e6e24166a525517702f146bd2a2610
Copying blob sha256:07e3d5e62bb2e04dc15e65e75d4993ceecaf3f0f7ad05a3d38ac491f173d5bec
Copying blob sha256:978b9e9305429b27a63a1e040a0924161a78d0d1109352a5173ccee50a9a077e
Copying blob sha256:07e3d5e62bb2e04dc15e65e75d4993ceecaf3f0f7ad05a3d38ac491f173d5bec
Copying config sha256:f705f21038d2a5c0136ee0ace7f5b9854fc0090767e42804b09ad7d384a73758
Writing manifest to image destination
Storing signatures
Error: statfs /Users/anarang/work/security_testing/rapidast/scripts: no such file or directory
exit code: 125
podman start -a zaproxy
Error: no container with name or ID "zaproxy" found: no such container
exit code: 125
anarang@anarang-mac:~/work/security_testing/rapidast (master %) $ ls                  
LICENSE               config                podman-compose-ui.yml policies              runenv-ui.sh          scripts               webswing
README.md             docker-compose.yml    podman-compose.yml    results               runenv.sh             test
anarang@anarang-mac:~/work/security_testing/rapidast (master %) $ podman start -a zaproxy                                                                        
Error: no container with name or ID "zaproxy" found: no such container
anarang@anarang-mac:~/work/security_testing/rapidast (master %) $ podman start -a zap2docker-stable                                                                                                                                                                         125
Error: no container with name or ID "zap2docker-stable" found: no such container

[anarang]

Adding an update here:
@jeremychoi mentioned that they investigated a little bit and it seems like the upstream zaproxy image we are using has an issue with the M1 chip. Linking the issue here zaproxy/zaproxy#6944 .

[darunesh-RH]

@anarang @jeremychoi can you please try pulling the below docker image and try to run zap on Mac M1, I made few changes and zap is working fine for me on M1 arch.

docker pull usr7771/zap-mac-m1:zap-mac-m1
docker run -it usr7771/zap-mac-m1:zap-mac-m1

[anarang]

@darunesh-RH thanks for taking the time to look into this. I pulled the image specified above.
So, here is the change I did to docker-compose.yml in rapidast repo and it seems to be working for me atm, however just the basic run:

version: "3.8"
services:
  zaproxy:
    env_file: .env
#    image: owasp/zap2docker-stable
    image: usr7771/zap-mac-m1:zap-mac-m1
    container_name: zaproxy

 $ docker-compose up zaproxy      
Creating zaproxy ... done
Attaching to zaproxy
zaproxy       | Collecting jinja2
zaproxy       |   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
zaproxy       | Collecting MarkupSafe>=2.0
zaproxy       |   Downloading MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (26 kB)
zaproxy       | Installing collected packages: MarkupSafe, jinja2
zaproxy       | Successfully installed MarkupSafe-2.1.1 jinja2-3.1.2
zaproxy       | Found Java version 11.0.15
zaproxy       | Available memory: 7951 MB
zaproxy       | Using JVM args: -Xmx1987m
zaproxy       | 352 [main] INFO  org.parosproxy.paros.Constant - Copying default configuration to /home/zap/.ZAP/config.xml
.....

Great to see that it worked on mac m1 after multiple failed attempts. ! woot, woot!

# Updated the config.yml with target url and open api url
$ docker-compose exec zaproxy python /zap/scripts/apis_scan.py may4_reports                                                                           
WARNING:root:The policy API-minimal-example.policy was already in ZAP. No modification were applied to the existing policy
$ cd may4_reports 
$ ls
demo-report-20220504-124403.xml sessions

I do see a xml report output here. @jeremychoi how do we get the html reports too after the scan is complete?

$ docker-compose down                                                                                                                                                                               130
Stopping zaproxy ... done
Removing zaproxy ... done
Removing network rapidast_default

Thanks again @darunesh-RH for the pointers here, I would suggest to raise a PR with the changes in Dockerfile. I am sure there are more people trying to solve this problem on their mac m1s. thanks.

[anarang]

Along with that, I do see that even though the container is running as expected it always lists the state as unhealthy. Any clue on that?

$ docker ps -a
CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                      PORTS                    NAMES
c7153760b24e   usr7771/zap-mac-m1:zap-mac-m1   "/zap/scripts/entryp…"   49 minutes ago   Up 49 minutes (unhealthy)   0.0.0.0:8091->8090/tcp   zaproxy

[jeremychoi]

zaproxy/zaproxy#7277 (comment) should solve the M1 issue.