"socket connect failed" but only if starting pigpiod via systemctl

[ralphhughes]

Not sure if I'm using it wrong but I can't seem to get pigpiod working if I start it as a service via systemd. Log of my actions as follows:

Made sure pigpiod was not running and then started it via systemctl:

pi@rpi-robot:~ $ ps aux | grep pigpiod
pi        1381  0.0  0.5   4364  1912 pts/0    S+   14:47   0:00 grep --color=auto pigpiod

pi@rpi-robot:~ $ sudo systemctl start pigpiod

pi@rpi-robot:~ $ pigs hwver
socket connect failed

Noticed that it seemed to have only bound to the ipv6 address ::1?

pi@rpi-robot:~ $ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      447/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      447/sshd
tcp6       0      0 ::1:8888                :::*                    LISTEN      1390/pigpiod
udp        0      0 0.0.0.0:1900            0.0.0.0:*                           436/minissdpd
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           244/avahi-daemon: r
udp        0      0 0.0.0.0:34300           0.0.0.0:*                           244/avahi-daemon: r
udp        0      0 0.0.0.0:68              0.0.0.0:*                           429/dhcpcd
udp6       0      0 :::42667                :::*                                244/avahi-daemon: r
udp6       0      0 :::5353                 :::*                                244/avahi-daemon: r

Killed that version and started up via sudo and it works fine. Notice how the bind address is different but the port (8888) is the same

pi@rpi-robot:~ $ sudo killall pigpiod
pi@rpi-robot:~ $ sudo pigpiod
pi@rpi-robot:~ $ pigs hwver
9437377
pi@rpi-robot:~ $ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      447/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      447/sshd
tcp6       0      0 :::8888                 :::*                    LISTEN      1428/pigpiod
udp        0      0 0.0.0.0:1900            0.0.0.0:*                           436/minissdpd
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           244/avahi-daemon: r
udp        0      0 0.0.0.0:34300           0.0.0.0:*                           244/avahi-daemon: r
udp        0      0 0.0.0.0:68              0.0.0.0:*                           429/dhcpcd
udp6       0      0 :::42667                :::*                                244/avahi-daemon: r
udp6       0      0 :::5353                 :::*                                244/avahi-daemon: r
pi@rpi-robot:~ $

This is pretty much a fresh install of Raspbian Lite 9 (Stretch) on a Pi Zero W.

[joan2937]

I'm afraid systemctl is a black art as far as I am concerned. I use the raspberrypi.org supplied service file (as installed with Raspbian full). I have no real understanding of how it works.

Perhaps someone else can help.

[ralphhughes]

The contents of the service file for reference:

[Unit]
Description=Daemon required to control GPIO pins via pigpio
[Service]
ExecStart=/usr/bin/pigpiod -l
ExecStop=/bin/systemctl kill pigpiod
Type=forking
[Install]
WantedBy=multi-user.target

So systemctl is running it slightly differently, passing the -l flag. But I'm trying to access it from localhost anyway so it should make no difference?

In pigs.c the default is to open a connection to PI_DEFAULT_SOCKET_ADDR_STR on PI_DEFAULT_SOCKET_PORT.

In pigpio.h these defaults are set to 127.0.0.1 and 8888.

Suspect this is caused by the latest version of Raspbian having ipv6 taking precedence over ipv4.

Test on older version of Raspbian (Raspbian Lite 8 Jessie)

$ sudo apt-get install pigpio
Reading package lists... Done
Building dependency tree
Reading state information... Done
pigpio is already the newest version

pi@rpi-desktop $ sudo netstat -tulpn | grep pigpio
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      418/pigpiod

$ pigs hwver
10489921

So weirdly even though I've started this via systemd, it seems to have bound to all ipv4 addresses? That explains why it works anyway.

So in summary:

In Raspbian 8, by default pigpiod binds to the ipv4 address for localhost on port 8888 and pigs by default tries to connect to it on the ipv4 address for localhost on port 8888 so it works fine.

In Raspbian 9, by default pigpiod binds to the ipv6 address for localhost on 8888, pigs still tries to connect to the ipv4 address by default and fails since 127.0.0.1 =/= ::1

Suspect that setting the address in the environment will fix this but I've run out of lunch break so I'll try it tomorrow.

[guymcswain]

Thanks for discovering the issue, debugging it and hopefully resolving it! I do not plan to move my hosts running pigpiod (also using systemd service) to Raspbian9 until there is a good reason to do so. In your case, is it just starting fresh with the latest OS or was there something you were avoiding on Raspbian8?

[schlimmchen]

@ralphhughes is perfectly right!

As a quick fix, I suggest to change the ExecStart stanza to:

ExecStart=/usr/bin/pigpiod -l -n 127.0.0.1

This makes the daemon bind to IPv4.

The longterm solution should be to make the daemon listen to both IPv4 and IPv6, I guess.

[guymcswain]

Tagging this with W/A and Retest since later versions of pigpio have altered the handling of localhost IP.

[guymcswain]

@maxnet ,
It appears that your contribution, PR #115, is only accepting IPv6 loopback address when -l option is applied to the daemon (pigpiod). I believe your intent was to handle IPv6 and IPv4 seamlessly from the application's perspective. In this case, any client written to connect to 'localhost'/127.0.0.1 will not be allowed to connect. Can you please provide a patch or let us know your status otherwise? Thank you.

[maxnet]

It appears that your contribution, PR #115, is only accepting IPv6 loopback address when -l option
is applied to the daemon (pigpiod).

Ah, yeah.
Guess the pigpiod code should be changed to support multiple listen sockets.
Accepting both IPv4 and IPv6 with a single listen socket do is possible when using the "any" address, but for some reason I never understood not with a loopback address.

In this case, any client written to connect to 'localhost'/127.0.0.1 will not be allowed to connect.

Actually, "localhost" do should resolve to both ::1 and 127.0.0.1

$ host localhost
localhost has address 127.0.0.1
localhost has IPv6 address ::1

If a hostname resolves to multiple addresses, the client really should try connecting to all of them before giving up.
So if "localhost" (instead of an explicit "127.0.0.1") also fails, that should ideally be fixed on client as well.

[guymcswain]

My host resolves localhost to both 127.0.0.1 and ::1. Yet when I request to connect to 'localhost' it chooses (I suppose) to use the IPv4. I don't think the burden should be placed on the clients. You would be asking a lot of apps to change their code.

[guymcswain]

*clients == apps in this case.

[maxnet]

Yet when I request to connect to 'localhost' it chooses (I suppose) to use the IPv4.

The client actually has a for() loop that do is supposed to try all addresses:

https://github.com/joan2937/pigpio/blob/master/pigs.c#L134

Will take a look why it is not working, when I have time.
Unless someone else beats me to it.

[guymcswain]

@maxnet, I appreciate your willingness to look into this. However, my guidance is to try to solve this in pigpio.c where I think it belongs. If you 'fix' pigs then you'll need to fix the python module and the other clients of which I believe there are 5 or 6 in total.

[maxnet]

However, my guidance is to try to solve this in pigpio.c where I think it belongs.

Sorry, don't have time to rewrite the server code to support multiple listen sockets.

If you 'fix' pigs then you'll need to fix the python module and the other clients of which I believe there are 5 or 6 in total.

Doubt those have same bug.

[guymcswain]

Accepting both IPv4 and IPv6 with a single listen socket do is possible when using the "any" address, but for some reason I never understood not with a loopback address.

Then seems like a possible solution is to leave the socket bound to IPv6 "any" and in the addrAllowed() function, if gpioCfg.internals & PI_LOCALHOST_SOCK_IF then test for both IPv6 and IPv4 loopback addresses (if not already whitelisted).

I'd attempt this myself but I'm not experienced in the network stack enough to know how to map IPv4 to IPv6 and do the address comparison.

[maxnet]

If a hostname resolves to multiple addresses, the client really should try connecting to all of them
before giving up.

The connect code in client is fine.
pigpio that ships with Raspbian is an older version that has 127.0.0.1 instead of "localhost" as default in the .h though...
So problem should no longer appear when they update to newer.

That said.
Modifying the socket listening code to support multiple sockets may be useful for other things. Like support for systemd socket activiation.
May have a look at it some time, but not this week.

[guymcswain]

For those following this thread, a work-around for this issue is posted on the wiki here.

[guymcswain]

Confirmed this issue is resolved after upgrade to pigpio V73.

[btsimonh]

Note that on a fresh install of v79 (rpi3b, bullseye), using sudo apt-get install pigpio

sudo netstat -tulpn | grep pigpio
reveals pigpiod listening only on tcp6.

The workaround of changing /lib/systemd/system/pigpiod.service to contain
ExecStart=/usr/bin/pigpiod -l -n 127.0.0.1
and then running

sudo systemctl daemon-reload
sudo systemctl restart pigpiod

changes it to plain tcp (v4), but only listening on localhost.

Q: -n 0.0.0.0 causes it to error. Is there some way to make it listen on all IPv4 addresses? - i.e. to specify ipv4, but enable remote, without explicitly specifying the local IP address?

[guymcswain]

From the doc:

If the -n option is not used all addresses are allowed (unless overridden by the -k or -l options). Multiple -n options are allowed. If -k has been used -n has no effect. If -l has been used only -n localhost has any effect

What are the options on ExecStart=/usr/bin/pigpiod after a fresh install?

[btsimonh]

ahh...
I'm sure I ADDED the -n, so the original (created by apt-get install?) must have been
ExecStart=/usr/bin/pigpiod -l
Tonight I will retry
ExecStart=/usr/bin/pigpiod -n 0.0.0.0 (without the -l) and see if it allows remote access over ipv4.
It makes sense that the combination of -l and -n 0.0.0.0 would not work.... and it makes sense that the default apt install should only listen on local (but would be much better if it at least defaulted to local ipv4?)

When you 'Confirmed this issue is resolved after upgrade to pigpio V73' in 2020, what was/is your startup line, and what was it listening on?
Ideally, pigpiod should listen on both ipv4 and ipv6.... I did not dig into the code to see if it tries.

(p.s. I'm using your pigpio-client - nice and simple. Will fork and propose some extra examples soon, plus maybe a node-red node).

[btsimonh]

with just -n 0.0.0.0: :(

    Process: 1446 ExecStart=/usr/bin/pigpiod -n 0.0.0.0 (code=exited, status=1/FAILURE)
        CPU: 12ms
Jan 19 17:29:03 raspberrypi3b systemd[1]: Starting Daemon required to control GPIO pins via pigpio...
Jan 19 17:29:03 raspberrypi3b pigpiod[1446]: invalid -n option (0.0.0.0)

line 148 in pigpiod:
err = getaddrinfo(addrStr, portStr, &hints, &res);
effectively bars the -n option from specifying all interfaces.
If addrStr was compared to '0.0.0.0', and found equal, then set to null, then it would work - but I'm not sure what then governs ipv4 vs ipv6.

if I specify -n 255.255.255.255 then

sudo netstat -tulpn | grep pigpio
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      1583/pigpiod

but pigpio-client does not see it.

If I use just ExecStart=/usr/bin/pigpiod then

sudo netstat -tulpn | grep pigpio
tcp6       0      0 :::8888                 :::*                    LISTEN      1653/pigpiod

and a local pigpio-client works. I have not tested a remote client.

Note that as originally installed with the -l option:

sudo netstat -tulpn | grep pigpio
tcp6       0      0 ::1:8888                :::*                    LISTEN      1720/pigpiod

pigpio-client cannot connect:

Unable to connect to pigpiod.  No retry timeout option was specified.  Verify that daemon is running from localhost:8888.
MyError [pigpioClientError]: connect ECONNREFUSED 127.0.0.1:8888

So, it seems that the -l option makes it only bind to ipv6 on the local address.

Conclusions:

1/ the default apt-get install ExecStart=/usr/bin/pigpiod -l only binds to ipv6 - and so is complicated for 'normal' users.

2/ the -n option cannot be used to cause it to bind to both ipv4 and ipv6.

3/ the default apt-get install should probably be ExecStart=/usr/bin/pigpiod -l -n 127.0.0.0 - since all local clients will probably try ipv4?

4/ ExecStart=/usr/bin/pigpiod - i.e. with no options - appears to bind to both ipv4 and ipv6. I cannot at the moment test remote connection from a client, but kitty (ipv4) behaved differently for no options vs the -l option, and netstat indicates :::8888 for no options, which to me indicates ipv6, so maybe it binds to both?

Whooo... and -n ::: crashes and burns badly - bug in linux?!!! - sudo systemctl restart pigpiod hangs until ctrl-c

[guymcswain]

When you 'Confirmed this issue is resolved after upgrade to pigpio V73' in 2020, what was/is your startup line, and what was it listening on?

No, it has been too long ago.

Ideally, pigpiod should listen on both ipv4 and ipv6....

I generally agree with that statement.

[rgobbel]

The problem is that the service has to be started by root. Adding:

User=root

before the ExecStart line fixes it for me.