UNKNOWN: Exception raised during check: TypeError('string indices must be integers',)

[matt-horwood-mayden]

Afternoon,

I have installed your very lovely tool via pip, but I seem to get an error UNKNOWN: Exception raised during check: TypeError('string indices must be integers',).

This is run on raspbian 9.11 using Python 3.5.3
the full output

matt@pi-docker01:~$ check_swarm  --service all
OK: Replicated service portainer_portainer OK; OK: Replicated service phpipam_phpipam OK; OK: Replicated service phpmyadmin_phpmyadmin OK; OK: Replicated service proxy_proxyv2 OK; OK: Replicated service authelia_openldap OK; OK: Replicated service authelia_smtp OK; OK: Replicated service gogs_gogs OK; OK: Replicated service jukebox_jukebox OK; OK: Replicated service pihole_pihole OK; OK: Replicated service dashmachine_dashmachine OK; UNKNOWN: Exception raised during check: TypeError('string indices must be integers',)

Full service list

ID                  NAME                      MODE                REPLICAS            IMAGE                               PORTS
ag3gcvplpwcp        authelia_authelia         replicated          1/1                 authelia/authelia:4.14.0            *:9091->9091/tcp
mr82j6w87o38        authelia_openldap         replicated          1/1                 osixia/openldap:1.3.0               *:389->389/tcp, *:636->636/tcp
1vygoxc8pv6b        authelia_phpldapadmin     replicated          1/1                 osixia/phpldapadmin:0.9.0           
wkl8csavuj0a        authelia_redis            replicated          1/1                 redis:latest                        
6oh5h4u1rj2q        authelia_smtp             replicated          1/1                 mhzawadi/postfix:v0.0.1             *:25->25/tcp
7tc5mrv9835w        dashmachine_dashmachine   replicated          1/1                 supermamon/rpi-dashmachine:latest   
h9zcnc9qbcmu        gogs_gogs                 replicated          1/1                 gogs/gogs-rpi:latest                *:2250->22/tcp
wfjs3mgcsbgr        jukebox_jukebox           replicated          1/1                 mhzawadi/subsonic_jukebox:v0.0.12   
wjlnmi9sxhe7        phpipam_phpipam           replicated          1/1                 mhzawadi/phpipam:v1.4.0.3           
rgciy0pvvhmo        phpmyadmin_phpmyadmin     replicated          1/1                 mhzawadi/phpmyadmin:v5.0.1.3        
2acbxhzku4q0        pihole_pihole             replicated          1/1                 pihole/pihole:latest                *:53->53/tcp, *:53->53/udp, *:67->67/udp
l3j9bx3cxe3u        portainer_agent           global              2/2                 portainer/agent:latest              
sz4ssl200ufh        portainer_portainer       replicated          1/1                 portainer/portainer:latest          *:9000->9000/tcp
x78edrwv5opb        proxy_proxyv2             replicated          1/1                 traefik:v2.0                        *:80->80/tcp, *:443->443/tcp, *:8181->8080/tcp

Docker version

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:37:36 2020
 OS/Arch:           linux/arm
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:31:37 2020
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[timdaman]

Thank you for the bug report. I haven't seen that one before.

I just realized there is a line of debugging code in check_docker that is missing from the check_swarm command.

Sorry to ask but can you please edit that command to insert the missing debugging line and send me the resulting output.

I have a patched script here.
https://github.com/timdaman/check_docker/raw/swarm_debugging/check_docker/check_swarm.py
You can download it and run it like so python3.5 check_swarm.py --service '.*'.

If you would prefer you can also make the change manually on the script you already have
Change check_swarm line 348-349 from

        except Exception as e:
            unknown("Exception raised during check: {}".format(repr(e)))

to

        except Exception as e:
            import traceback
            traceback.print_exc()
            unknown("Exception raised during check: {}".format(repr(e)))

Sorry about that.

[matt-horwood-mayden]

Hi @timdaman
have run your change and get the below error, hope that helps track down the issue

Traceback (most recent call last):
  File "check_swarm.py", line 347, in perform_checks
    check_service(name=service, ignore_paused=args.ignore_paused)
  File "check_swarm.py", line 226, in check_service
    process_global_service(name=name, ignore_paused=ignore_paused)
  File "check_swarm.py", line 196, in process_global_service
    if task['Status']['State'] != 'running':
TypeError: string indices must be integers
OK: Replicated service authelia_redis OK; OK: Replicated service dashmachine_dashmachine OK; OK: Replicated service gogs_gogs OK; UNKNOWN: Exception raised during check: TypeError('string indices must be integers',)

[timdaman]

Oh my, I think I may have found the cause. I think was a bug in how global services were being processed. I have another update for you to try(same url as before).
https://github.com/timdaman/check_docker/raw/swarm_debugging/check_docker/check_swarm.py
Can you let me know if that fixes your error.

[matt-horwood-mayden]

Hi @timdaman
Yep, that did it no error.

OK: Replicated service phpipam_phpipam OK; OK: Replicated service authelia_redis OK; OK: Replicated service authelia_openldap OK; OK: Replicated service gogs_gogs OK; OK: Replicated service jukebox_jukebox OK; OK: Replicated service phpmyadmin_phpmyadmin OK; OK: Replicated service authelia_phpldapadmin OK; OK: Global service portainer_agent OK; OK: Replicated service proxy_proxyv2 OK; OK: Replicated service authelia_authelia OK; OK: Replicated service pihole_pihole OK; OK: Replicated service dashmachine_dashmachine OK; OK: Replicated service portainer_portainer OK; OK: Replicated service authelia_smtp OK

[timdaman]

Thank you for the bug report and the help debugging. I have just shipped v. 2.2.2 which contains the fix.