internal/github module causes crash after some period of time

[apetresc]

I have the following module configured in my polybar config:

[module/github]
type = internal/github
token = xxxxxxxxxx
empty-notifications = false
format = <label>
label = %{A1:xdg-open https\://github.com/notifications/:}  %notifications% %{A}
format-underline = #3f4448

(Yes, my polybar is compiled with +curl)

If I add this module to a bar, everything appears to work fine, I can see the entry, it refreshes every so often, etc. However, randomly and at uneven intervals, but always within ~24 hours, polybar will crash without any sort of output spilled to stderr or stdout.

Even stranger, it doesn't even happen at the same time if I have multiple bars (one for each monitor). Maybe after 3 hours one monitor will lose polybar, and then 10 hours later, the other monitor will lose it.

Removing github from the bar makes everything return to normal.

Any hints on how I can start diagnosing this issue? Again, I don't see anything on stdout/stderr. Should I try to get a core dump or something? Has anyone encountered this before?

[patrick96]

The first step in debugging this would probably be running polybar with the -l info and posting the output of that.
Also, when polybar terminates, what is its return code?

[apetresc]

Cool, thanks – I will grab both of those pieces of info for you right away. It just might take a few hours for the crash to happen :)

[apetresc]

Okay, both have crashed now. In both cases, there was nothing remarkable in the output, even with -l info (just a bunch of 'redrawing bar window`s, etc), and the exit code in both cases was 141.

[patrick96]

Hmm, the return code seems to suggest that polybar terminated due to SIGPIPE.
Can you confirm for me that the github module is actually the culprit by removing all other modules from the modules-* lists.

While you're at it, you can start debugging with gdb. You'll have to recompile polybar with debug symbols (add -DCMAKE_BUILD_TYPE=Debug to cmake), then run the bar and attach to the process using sudo gdb -p PID -ex cont | tee /path/to/logfile. Once the crash occurs, gdb should show a message along the lines of

Thread 1 "polybar" received signal SIGPIPE, Broken pipe

and the bar will stop updating.
Once that happens, run info stack in gdb and then post the contents of the log file here.
If you don't know how to do some of those things, please tell me and I'll write up a quick guide

[apetresc]

No problem - yes, I know how to do all those things :) I will give it a shot tonight, but given the unpredictable nature of the crash, it might be tomorrow by the time I get you those logs. Thanks again for the help!

…

On Fri, Sep 15, 2017 at 6:52 PM, Patrick Ziegler ***@***.***> wrote: Hmm, the return code seems to suggest that polybar terminated due to SIGPIPE. Can you confirm for me that the github module is actually the culprit by removing all other modules from the modules-* lists. While you're at it, you can start debugging with gdb. You'll have to recompile polybar with debug symbols (add -DCMAKE_BUILD_TYPE=Debug to cmake), then run the bar and attach to the process using sudo gdb -p PID -ex cont | tee /path/to/logfile. Once the crash occurs, gdb should show a message along the lines of Thread 1 "polybar" received signal SIGPIPE, Broken pipe and the bar will stop updating. Once that happens, run info stack in gdb and then post the contents of the log file here. If you don't know how to do some of those things, please tell me and I'll write up a quick guide — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#754 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA6GJTG-3zuBQQ996QHacoOEzuqWSmbks5siv-4gaJpZM4PZDaT> .

[apetresc]

Okay, interesting development - I had been running the newest released version in Arch AUR (3.0.5-4), but I switched to git master for building with debug symbols. And now on master, instead of the whole bar crashing after a few hours, just the GitHub module disappears (until I restart the bar, and then it reappears). So something relevant changed between polybar and polybar-git on AUR...

[patrick96]

This is very strange, I have looked at the changes made since v3.0.5 and none of them should impact the github module.
Does polybar give you an error, once the github module disappears?

Can you still debug version 3.0.5, it might still help to track down the issue

[patrick96]

While reviewing #811 I have been able to reproduce the crash. I don't think anything has changed between version 3.0.5 and master but rather both variants sometimes occur (crash and module dissappearance). If curl returns a non 200 return code, the module throws an exception and polybar disables it. This is fixed by #811. The crash occurs only if a SIGPIPE signal is received somewhere. From my testing I have gathered the following stack trace at the point the SIGPIPE signal was received:

#0  0x00007ffff4c618bb in write () from /usr/lib/libpthread.so.0
#1  0x00007ffff1a04357 in ?? () from /usr/lib/libcrypto.so.1.1
#2  0x00007ffff19fee75 in BIO_write () from /usr/lib/libcrypto.so.1.1
#3  0x00007ffff1e0058e in ?? () from /usr/lib/libssl.so.1.1
#4  0x00007ffff1e0082c in ?? () from /usr/lib/libssl.so.1.1
#5  0x00007ffff1e09422 in ?? () from /usr/lib/libssl.so.1.1
#6  0x00007ffff1e07806 in ?? () from /usr/lib/libssl.so.1.1
#7  0x00007ffff1e11d60 in SSL_shutdown () from /usr/lib/libssl.so.1.1
#8  0x00007ffff708d7c6 in ?? () from /usr/lib/libcurl.so.4
#9  0x00007ffff708d831 in ?? () from /usr/lib/libcurl.so.4
#10 0x00007ffff7053a72 in ?? () from /usr/lib/libcurl.so.4
#11 0x00007ffff706641b in ?? () from /usr/lib/libcurl.so.4
#12 0x00007ffff7068c6b in ?? () from /usr/lib/libcurl.so.4
#13 0x00007ffff706a0c4 in curl_multi_perform () from /usr/lib/libcurl.so.4
#14 0x00007ffff705fe39 in curl_easy_perform () from /usr/lib/libcurl.so.4
#15 0x0000555555a72683 in polybar::v3_0_5_74_ga0c6eab::http_downloader::get (this=0x555555e27350, url="https://api.github.com/notifications?access_token=[redacted]") at /home/patrick96/Projects/github.com/kokan/polybar/src/utils/http.cpp:30
#16 0x00005555559eb63e in polybar::v3_0_5_74_ga0c6eab::modules::github_module::request[abi:cxx11]() (this=0x555555e28e90) at /home/patrick96/Projects/github.com/kokan/polybar/src/modules/github.cpp:45
#17 0x00005555559eb982 in polybar::v3_0_5_74_ga0c6eab::modules::github_module::get_number_of_notification (this=0x555555e28e90) at /home/patrick96/Projects/github.com/kokan/polybar/src/modules/github.cpp:65
#18 0x00005555559eb570 in polybar::v3_0_5_74_ga0c6eab::modules::github_module::update (this=0x555555e28e90) at /home/patrick96/Projects/github.com/kokan/polybar/src/modules/github.cpp:36
#19 0x00005555559eda1a in polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::runner()::{lambda()#1}::operator()() const (__closure=0x7fffe7ffebb0) at /home/patrick96/Projects/github.com/kokan/polybar/include/modules/meta/timer_module.hpp:25
#20 0x00005555559edb79 in polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::runner (this=0x555555e28e90) at /home/patrick96/Projects/github.com/kokan/polybar/include/modules/meta/timer_module.hpp:34
#21 0x00005555559ee0f4 in std::__invoke_impl<void, void (polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::*)(), polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>*> (__f=@0x555555e24510: (void (polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::*)(polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module> * const)) 0x5555559eda62 <polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::runner()>, __t=@0x555555e24508: 0x555555e28e90) at /usr/include/c++/7.2.0/bits/invoke.h:73
#22 0x00005555559edd3f in std::__invoke<void (polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::*)(), polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>*> (__fn=@0x555555e24510: (void (polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::*)(polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module> * const)) 0x5555559eda62 <polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::runner()>, __args#0=@0x555555e24508: 0x555555e28e90) at /usr/include/c++/7.2.0/bits/invoke.h:95
#23 0x00005555559ee3b3 in std::thread::_Invoker<std::tuple<void (polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::*)(), polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>*> >::_M_invoke<0ul, 1ul> (this=0x555555e24508) at /usr/include/c++/7.2.0/thread:234
#24 0x00005555559ee354 in std::thread::_Invoker<std::tuple<void (polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::*)(), polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>*> >::operator() (this=0x555555e24508) at /usr/include/c++/7.2.0/thread:243
#25 0x00005555559ee324 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>::*)(), polybar::v3_0_5_74_ga0c6eab::modules::timer_module<polybar::v3_0_5_74_ga0c6eab::modules::github_module>*> > >::_M_run (this=0x555555e24500) at /usr/include/c++/7.2.0/thread:186
#26 0x00007ffff548ba6f in std::execute_native_thread_routine (__p=0x555555e24500) at /build/gcc-multilib/src/gcc/libstdc++-v3/src/c++11/thread.cc:83
#27 0x00007ffff4c5808a in start_thread () from /usr/lib/libpthread.so.0
#28 0x00007ffff499024f in clone () from /usr/lib/libc.so.6

The SIGPIPE is received somewhere inside the call to curl, so there is not much we can do about it other than ignore the SIGPIPE signals. From the reading I did, we can do that by calling the following function:

signal(SIGPIPE, SIG_IGN);

The thing with this is that it will now ignore all the SIGPIPE signals, not just the ones from the curl call. Optimally we would want to temporarily ignore the signal only for the curl call. Also I think signal(...) may not work properly with multiple threads.
@NBonaparte do you have an idea how we could handle this?

[NBonaparte]

curl is supposed to ignore sigpipes in curl_easy_perform() and curl_multi_perform(), as it calls an internal function sigpipe_ignore() in its source (see lib/sigpipe.h).

For signal(), we can restore the default handling by calling signal(SIGPIPE, SIG_DFL).

If we want to use sigaction(), we can follow sigpipe_ignore() and sigpipe_restore() from lib/sigpipe.h and see if that fixes the issue.

[apetresc]

I've been running master for about 2 days now and haven't had it crash with #811 applied yet. So probably the main source of SIGPIPEs has been addressed, and you're looking for a more granular way of dealing with unexpected ones, is that right? In that case, should I keep this issue open to track that, or shall I close it?

[patrick96]

We probably should leave this one open.
The module disappearing forever was not caused by a SIGPIPE, but rather by the module throwing an uncaught exception whenever curl failed which in turn made the controller disable the module (this is what was fixed in #811). The original issue here was the crash, which still can occur and is caused by SIGPIPE. I will play around with the function mentioned by NBonaparte and see what fixes the issue.

[saneki]

This issue was referenced in 3.1.0 but isn't fixed, as SIGPIPE still isn't caught or ignored. There seems to already be code for this in src/adapters/mpd.cpp, so I'm wondering if enabling the mpd module will prevent the crash from happening. However, that code is used to determine whether it is connected to the server or not.

@NBonaparte The curl SIGPIPE functions could work, but only if there's no risk of one module receiving a SIGPIPE (that it acts on) while another module is between sigpipe_ignore and sigpipe_restore. So single-threaded only.

[patrick96]

The reference to this issue seems to be an error, it was supposed to point to 744 not 754. I have fixed that in release notes.

I'll have to test out, if an enabled mpd module will prevent this