Failure on Ansible Task Ensure internet-monitoring environment is running - Grafana Failure

[KunalGehlot]

I'm facing a failure on the specified task, I'm running the latest Raspberry Pi OS and I noticed on Grafana Docker Hub that they don't have an image for linux/arm/v8.

I believe forcing docker to install v7 should work but I'm not sure if that's correct and where I can do that within the ansible Task.

Here's the detailed traceback:

TASK [Ensure internet-monitoring environment is running.] ********************************************************************************************************************************************************
task path: /home/zack/internet-pi/tasks/internet-monitoring.yml:47
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: zack
<127.0.0.1> EXEC /bin/sh -c 'echo ~zack && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/zack/.ansible/tmp `"&& mkdir "` echo /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369 `" && echo ansible-tmp-1696255007.7593706-5892-48847898244369="` echo /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369 `" ) && sleep 0'
Using module file /home/zack/.local/lib/python3.9/site-packages/ansible_collections/community/docker/plugins/modules/docker_compose.py
<127.0.0.1> PUT /home/zack/.ansible/tmp/ansible-local-5349i998sglj/tmp3ugo1lxe TO /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369/AnsiballZ_docker_compose.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369/ /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369/AnsiballZ_docker_compose.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369/AnsiballZ_docker_compose.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/zack/.ansible/tmp/ansible-tmp-1696255007.7593706-5892-48847898244369/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
  File "/tmp/ansible_community.docker.docker_compose_payload_ol_b5jul/ansible_community.docker.docker_compose_payload.zip/ansible_collections/community/docker/plugins/modules/docker_compose.py", line 861, in cmd_up
  File "/usr/local/lib/python3.9/dist-packages/compose/project.py", line 664, in up
    svc.ensure_image_exists(do_build=do_build, silent=silent, cli=cli)
  File "/usr/local/lib/python3.9/dist-packages/compose/service.py", line 358, in ensure_image_exists
    self.pull(silent=silent)
  File "/usr/local/lib/python3.9/dist-packages/compose/service.py", line 1251, in pull
    return progress_stream.get_digest_from_pull(event_stream)
  File "/usr/local/lib/python3.9/dist-packages/compose/progress_stream.py", line 99, in get_digest_from_pull
    for event in events:
  File "/usr/local/lib/python3.9/dist-packages/compose/service.py", line 1223, in _do_pull
    yield from stream_output(output, sys.stdout)
  File "/usr/local/lib/python3.9/dist-packages/compose/progress_stream.py", line 27, in stream_output
    print_output_event(event, stream, is_terminal)
  File "/usr/local/lib/python3.9/dist-packages/compose/progress_stream.py", line 59, in print_output_event
    raise StreamOutputError(event['errorDetail']['message'])
fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "errors": [],
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "build": false,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "debug": false,
            "definition": null,
            "dependencies": true,
            "docker_host": "unix://var/run/docker.sock",
            "env_file": null,
            "files": null,
            "hostname_check": false,
            "nocache": false,
            "profiles": null,
            "project_name": null,
            "project_src": "/home/zack/internet-monitoring/",
            "pull": false,
            "recreate": "smart",
            "remove_images": null,
            "remove_orphans": false,
            "remove_volumes": false,
            "restarted": false,
            "scale": null,
            "services": null,
            "ssl_version": null,
            "state": "present",
            "stopped": false,
            "timeout": null,
            "tls": false,
            "tls_hostname": null,
            "use_ssh_client": false,
            "validate_certs": false
        }
    },
    "module_stderr": "",
    "module_stdout": "latest: Pulling from grafana/grafana\n",
    "msg": "Error starting project no matching manifest for linux/arm/v8 in the manifest list entries"
}

[KunalGehlot]

Looks like I was on the wrong track, I did force docker to use v7 by adding a line in /internet-pi/templates/docker-compose.yml.j2 - platform: linux/arm/v7.

But now it says Error starting project no matching manifest for linux/arm/v7 in the manifest list entries.

P.S: Turns out grafana docker hub does not have the ARM build for the latest tag.

[benlbaum]

Kunal, I'm getting the same thing.

TASK [Ensure internet-monitoring environment is running.] **************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "errors": [], "module_stderr": "", "module_stdout": "latest: Pulling from miguelndecarvalho/speedtest-exporter\nDigest: sha256:f1064d49124c7fc45faabb87c6c876a2fd04e92b3dc14d4b871301217ba30fed\nStatus: Downloaded newer image for miguelndecarvalho/speedtest-exporter:latest\nlatest: Pulling from prom/prometheus\nDigest: sha256:c5dd3503828713c4949ae1bccd1d8d69f382c33d441954674a6b78ebe69c3331\nStatus: Downloaded newer image for prom/prometheus:latest\nlatest: Pulling from grafana/grafana\n", "msg": "Error starting project no matching manifest for linux/arm/v8 in the manifest list entries"}

It's driving me nuts.

[benlbaum]

Here is a little on the issue

`pi@internetpi1:~ $ cd internet-monitoring/
pi@internetpi1:~/internet-monitoring $ docker-compose up -d
Pulling grafana (grafana/grafana:latest)...
ERROR: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)`

[benlbaum]

Also this

pi@internetpi1:~/internet-monitoring $ docker-compose up -d
Pulling grafana (grafana/grafana:latest)...
latest: Pulling from grafana/grafana
ERROR: no matching manifest for linux/arm/v8 in the manifest list entries

[benlbaum]

Alright I've learned something. This seems to be an issue with the docker-compose.yml file pulling the latest version of Grafana and Grafana not having the linux/arm/v8 in their manifest. I tried changing latest to a few version back of Grafana and that didnt work either, so I went like a year back and tried 9.4.2. That fixed it! I don't actually know where the cut-off version is, but I'll test more later. For now here's how you can get it running.

cd ~/internet-pi/templates
nano docker-compose.yml.j2

Change grafana/grafana:latest to grafana/grafana:9.2.4

feel free to play around and find the most recent release that still works. You can find the releases here: https://github.com/grafana/grafana/releases

[benlbaum]

Well, that worked to get just blank grafana running using the ~/internet-monitoring/ folder and docker-compose up -d, but it is not working inside of the ansible-playbook... so idk and I've got to go to bed unfortunately

[benlbaum]

Wait, it actually works i think... but you have to do some weird stuff. change the docker-compose.yml.j2 file like I said above. Then run the playbook. This will copy the yml.j2 file to ~/internet-monitoring/docker-compose.yml. The play book will get stuck at the Internet Monitoring Check again. Ctrl+C out of it. Then cd ~/internet-monitoring/. Then docker-compose up -d. Then back to cd ~/internet-pi/. Run ansible-playbook main.yml. Everything should succeed this time.

Finally, go to your pi's IP Address :3030. Then click the magnifying glass for search dashboards. Click the General folder then the Internet Connection dashboard. Star it for easier access. Then I think you're good. What a PITA! Glad I got it, I'll check back tomorrow for connection logs.

[netresident]

Thanks a lot @benlbaum !
Tried 10.1.2 and it worked! listed here as the latest release with armv7 support: https://hub.docker.com/r/grafana/grafana/tags

[KunalGehlot]

Wait, it actually works i think... but you have to do some weird stuff. change the docker-compose.yml.j2 file like I said above. Then run the playbook. This will copy the yml.j2 file to ~/internet-monitoring/docker-compose.yml. The play book will get stuck at the Internet Monitoring Check again. Ctrl+C out of it. Then cd ~/internet-monitoring/. Then docker-compose up -d. Then back to cd ~/internet-pi/. Run ansible-playbook main.yml. Everything should succeed this time.

I'm close, but I'm getting the following error on docker-compose up -d

network internet-monitoring-back-tier was found but has incorrect label com.docker.compose.network set to "internet-monitoring-back-tier"

Update:

Looks like Docker is not reading the custom network names, change the compose file to

networks:
  internet-monitoring-front-tier:
  internet-monitoring-back-tier:

and updated the network reference accordingly.

But now it says

Error response from daemon: driver failed programming external connectivity on endpoint internet-monitoring-nginx-proxy-1 (b69305b73c74f0cd0cc0f1e7c7cb41f4d0ba4a5b1d461e61d7257ba48c6ca805): Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use

[KunalGehlot]

Sidenote: When I run docker-compose up -d in ~/internet-monitoring/ I get the error

parsing /home/zack/internet-monitoring/docker-compose.yml: yaml: unmarshal errors:
  line 70: mapping key "depends_on" already defined at line 61

and in-fact, there is two mentions of depends_on under grafana, just combined it into one.

Update: This happens when you set domain_name_enable to true in config.yml

[KunalGehlot]

Going for a hard-reset of the whole system...

[KunalGehlot]

Update: It worked! After all the patches discussed above, and additionally changing become: true at pi-hole.yml at Line 24 did the trick.

Now the only problem remains is that I cannot log into Grafana :/

[KunalGehlot]

Realised, the devs might want to patch this, re-opening.

[seandb]

this worked! i was ready to give up. I'm in!!!

[jcreek]

nano docker-compose.yml.j2

Confirming this also worked for me :)

[debruyneOneTouch]

New to networking, not a SW developer. Raspberry Pi is a new hobby. Pihole is running successfully as a DNS and add blocker. Grafana is being used in conjuction with Teslamate. Having some internet issues so now trying to get internet-pi working.

Initially was getting the grafana manifest error. I changed to grafana: 9.2.4 and changed line in pi-hole.yml to become: true.

Now having a Pihole issue. Any suggestions? thanks

TASK [Ensure Pi-hole is running.] *****************************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "errors": [], "module_stderr": "Creating pihole ... \n\u001b[1A\u001b[2K\nCreating pihole ... \n\u001b[1B", "module_stdout": "", "msg": "Error starting project Encountered errors while bringing up the project."}

[debruyneOneTouch]

Solved: removed the pre-existing pihole container with docker rm command

Internet-pi working nicely

[github-actions]

This issue has been marked 'stale' due to lack of recent activity. If there is no further activity, the issue will be closed in another 30 days. Thank you for your contribution!

Please read this blog post to see the reasons why I mark issues as stale.

[github-actions]

This issue has been closed due to inactivity. If you feel this is in error, please reopen the issue or file a new issue with the relevant details.