Reboot held up by Crowdsec refusing to stop #4262

roens · 2024-09-28T00:32:19Z

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
I have searched the existing issues, open and closed, and I'm convinced that mine is new.
The title contains the plugin to which this issue belongs

Describe the bug
I've lost track of which release this started with (for me). I regret not bringing this up when it began. But the trouble is that rebooting OPNsense is held up by Crowdsec "refusing" to quit. Others have faced this this as well.

Specifically (and for me, most commonly), this occurs for updates requiring reboot. The update log reported to the webUI ends with something like:

[39/48] Upgrading crowdsec from 1.6.2_4 to 1.6.3_1...
[39/48] Extracting crowdsec-1.6.3_1: .......... done
crowdsec is running as pid 84792.
Stopping crowdsec.
Waiting for PIDS: 84792

It will hang there until I ssh in, and kill that PID, resulting in it proceeding with the updates and automated reboot.

To Reproduce
Steps to reproduce the behavior:

Install an update which requires reboot OR choose: Power -> Reboot -> Yes
Expect it to reboot: find it is not
Kill the Crowdsec processes:
- pkill -9 -f 'daemon: crowdsec'
- pkill -9 -f 'crowdsec -c' (this PID may also need to be whacked, but not always)
Not that the reboot proceeds (if held up for an update, the update log in webUI will resume progressing)
- … though sometimes it still won't and additionally requires a reboot from that same ssh session
- OTOH, if a reboot is attempted prior to killing Crowdsec, that will fail for waiting for Crowdsec to quit nicely, requiring step 3 above

Expected behavior
That a reboot triggered will have no problem killing/quitting all running processes, and finally resulting in rebooting the system.

Screenshots
n/a

Relevant log files
(none that I could find, aside from update log in webUI pasted above)

Additional context
Once this behavior started, it has been consistently like this, every time.

Environment

OPNsense 24.7.4_1-amd64
FreeBSD 14.1-RELEASE-p4
OpenSSL 3.0.15

System: Supermicro SYS-E302-9D
CPU: Intel(R) Xeon(R) D-2123IT CPU @ 2.20GHz
Network: Intel I350-AM4 & Intel X557

The text was updated successfully, but these errors were encountered:

Starkstromkonsument · 2024-09-28T08:15:19Z

Thanks for creating this issue. Same Problem here. It also happens if you try to stop or restart Crowdsec via the WebUI.

IMHO there are two Problems:

The Crowdsec stop routine does not work properly
OPNsense does not kill hanging processes during shutdown / reboot after a reasonable timeout --> should this be a separate issue at opnsense/core?

fichtner · 2024-09-28T08:51:58Z

I don’t agree with no 2. this is a sporadic issue that doesn’t have operational impact overall (as in breaks running operation of the firewall).

Cheers,
Franco

davesnthere · 2024-10-01T14:01:25Z

Having the same issue here too.
After the hotfix update (24.7.5_3) OPNsense will not reboot.
I tried stopping from the GUI but after a couple min, no change.
Tried disabling the first 3 option in the CrowdSec settings then attempt to stop service, no luck.
Just discovered, for what ever reason, I can not ssh into my system.
Will have to wait till I get home and on the local console to try pkill.

Update:
After pkilling the CrowdSec, I was able to reboot. After the system came back up I was till on 24.7.5 so I shutdown CrowdSec and did the update again. This time it took and rebooted without issues. After system came back up I restarted CrowdSec.

kumy · 2024-10-01T15:09:18Z

Facing the same issue. Anything you wanna test?

EDIT 1

# ps axuHd

root      1   0.0  0.0   11304   1100  -  ILs  16:33    0:00.04 - /sbin/init
[…]
root  64368   0.0  0.4 1468772 117036  -  S    16:39    0:00.57 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  S    16:39    0:00.99 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.09 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.54 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  S    16:39    0:00.72 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.00 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.45 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.11 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  S    16:39    0:00.61 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.10 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.16 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.10 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.14 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.09 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.06 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.09 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  S    16:39    0:00.52 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  S    16:39    0:00.31 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.44 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.01 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  I    16:39    0:00.01 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root  64368   0.0  0.4 1468772 117036  -  S    16:39    0:00.47 |-- /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
[…]
root  87329   0.0  0.0   13280   2860  -  Is   16:38    0:00.00 |-- /bin/sh /usr/local/opnsense/scripts/firmware/launcher.sh update
root  90581   0.0  0.0   12872   2456  -  I    16:38    0:00.00 | `-- /usr/local/bin/flock -n -o /tmp/pkg_upgrade.progress /usr/local/opnsense/scripts/firmware/update.sh
root  90591   0.0  0.0   13280   2860  -  I    16:38    0:00.00 |   `-- /bin/sh /usr/local/opnsense/scripts/firmware/update.sh
root  91329   0.0  0.0   13280   2476  -  I    16:40    0:00.00 |     `-- /bin/sh /usr/local/etc/rc.reboot
root  91412   0.0  0.0   13280   2500  -  I    16:40    0:00.00 |       `-- /bin/sh /usr/local/etc/rc.syshook stop
root  95898   0.0  0.0   13280   2476  -  I    16:40    0:00.00 |         `-- /bin/sh /usr/local/etc/rc.syshook.d/stop/80-freebsd
root  96122   0.0  0.0   13280   2612  -  I    16:40    0:00.02 |           `-- /bin/sh /usr/local/etc/rc.freebsd stop
root  58077   0.0  0.0   13280   2792  -  I    16:40    0:00.00 |             `-- /bin/sh /usr/local/etc/rc.d/crowdsec stop
root  60483   0.0  0.0   12620   2024  -  I    16:40    0:00.00 |               `-- pwait 64368

# kill 64368
*** FINAL System shutdown message from xxx ***              

System going down IMMEDIATELY                                                  

                                                                               
Connection to xxx closed by remote host.
Connection to xxx closed.

EDIT2:
problem is the same after the system upgrade, rc script still wait for process to finish

fichtner · 2024-10-01T15:14:55Z

Crowdsec made a barely visible effort here https://www.reddit.com/r/opnsense/comments/1fpt7xn/comment/lp3lixx/ -- but it would be nice to have someone on GitHub like in the old days ;)

RudiKlein · 2024-10-02T11:16:06Z

@fichtner I have been coping with this sporadic issue every upgrade. Today again.
Maybe lots of people are having this issue but use the workaround without reporting it.
It's not a big issue, just very annoying.

bye,

Rudi

fichtner · 2024-10-02T11:25:37Z

I agree and hope crowdsec fill fix the issue soon in their code.

Unspec7 · 2024-10-02T21:02:01Z

I too am facing this issue. Current workaround is to manually stop crowdsec BEFORE rebooting.

nhatlinh1982 · 2024-10-04T11:11:21Z

I am too facing this issue. Not sure how long it has been having since i do not reboot my OPNsense too often

daygle · 2024-10-05T06:00:42Z

Same issue here too. In fact I have been experiencing for some time now.

mmetc · 2024-10-07T13:41:09Z

Hi,

could you test this

# fetch -o /usr/local/etc/rc.d/crowdsec https://github.com/crowdsecurity/plugins/releases/download/crowdsec-1.6.3-2-hotfix/crowdsec
# killall crowdsec

and try start/stop.

Thanks

Edit: added the kill command

mmetc · 2024-10-07T13:47:19Z

Crowdsec made a barely visible effort here https://www.reddit.com/r/opnsense/comments/1fpt7xn/comment/lp3lixx/ -- but it would be nice to have someone on GitHub like in the old days ;)

Sorry for the lack of communication, I was not responsive enough across the board but you are right, github is the place I prefer as well. The issue page in crowdsecurity/crowdsec ensures attention from all the team.

By the way I'm thinking that a beta-test process is in order, because the plugin needs an overhaul and its user base has doubled in a few months.

RudiKlein · 2024-10-07T19:49:36Z

@mmetc I only observed the Crowdsec service (processes) hanging during system updates.

I have installed the hotfix.
Tried to stop the crowdsec processes (to make sure they were using the hotfix), but one of them didn't wanna leave.
Rebooted the system
Stopped and started crowdsec from the gui. That seemed to work.
Tried to check the process status from the CLI, but now SSH is giving me a hard time. (it's one of these days).

So, status unknown for now. I'll give it a go tomorrow. Today the users can't handle more downtime.

Rudi

daygle · 2024-10-07T23:53:49Z

I would like to share my testing related to the original issue.

I have found the below during testing: -

After a restart of OPNsense (by stopping Crowdsec manually before the restart). Then wait for OPNsense to boot, if you then perform another reboot, it will stop waiting for to stop (without restarting).
From the shell if you reboot, then receive the 'Waiting for PIDs' message select Ctrl+C then reboot again - it will reboot normally the second time.
Now the interesting thing is. If you, let's say restart OPNsense. Then, after OPNsense has started restart Crowdsec from the 'Services' section then reboot again the reboot will perform normally.
This can be replicated without the restart of OPNsense by running a shell 'service crowdsec stop' - you will receive the waiting for PIDs message. Please note point 1 - only after a reboot/without the Crowdsec services stopped/restarted.

nhatlinh1982 · 2024-10-08T01:39:32Z

As i said before I also was having problem with this issue but I have another issue. While I am not sure if it is related to crowdsec but I was start having random shutdown of OPNsense VM in Proxmox AND having reboot problem. Because I only recently installed it crowdsec plugin, AND updated to OPNsense 24.7 from 24.1. I rolled back to old backup and currently running 24.7 OPNsense without crowdsec with NO issue for 2 days. I will install crowdsec after a week if there are no random crash, and observe if crowdsec is causing this issue again.

daygle · 2024-10-08T01:57:04Z

@nhatlinh1982 Sounds like the same issue to me.

mmetc · 2024-10-08T07:21:59Z

I would like to share my testing related to the original issue.

I have found the below during testing: -
[...]

Thanks for testing! You can update the script with this command until 1.6.3-2 is out.

# fetch -o /usr/local/etc/rc.d/crowdsec https://github.com/crowdsecurity/plugins/releases/download/crowdsec-1.6.3-2-hotfix/crowdsec
# killall crowdsec

Edit: added killall

daygle · 2024-10-08T07:44:46Z

I would like to share my testing related to the original issue.
I have found the below during testing: -
[...]

Thanks for testing! You can update the script with this command until 1.6.3-2 is out.
# fetch -o /usr/local/etc/rc.d/crowdsec https://github.com/crowdsecurity/plugins/releases/download/crowdsec-1.6.3-2-hotfix/crowdsec

Thanks @mmetc - Great news, initial testing is showing that this fix has corrected the issue. Any chance you can have a look at the other issue with Crowdsec? Stop/Start button issue - #4280.

The system will reboot. Do you want to proceed? [y/N]: y

Invoking stop script 'beep'
Invoking stop script 'freebsd'
Stopping crowdsec_firewall.
Stopping php_fpm.
Waiting for PIDS: 20611.
Stopping crowdsec.
Waiting for PIDS: 77982.
Waiting for PIDS: 78051.
Stopping nginx.
Waiting for PIDS: 13630.
Stopping acme_http_challenge.
Waiting for PIDS: 3552.
Stopping suricata.
Waiting for PIDS: 99017.
Stopping flowd.
Waiting for PIDS: 93666 94298.
Stopping flowd_aggregate...done
Stopping monit.
Waiting for PIDS: 85721.
crowdsec not running? (check /var/run/crowdsec_daemon.pid).
crowdsec_firewall is not running.
INFO/keactrl: Stopping kea-dhcp4...
INFO/keactrl: kea-dhcp6 isn't running.
INFO/keactrl: kea-dhcp-ddns isn't running.
INFO/keactrl: kea-ctrl-agent isn't running.
Invoking stop script 'backup'
Invoking backup script 'captiveportal'
Invoking backup script 'dhcpleases'
Invoking backup script 'duid'
Invoking backup script 'netflow'
Invoking backup script 'rrd'
Invoking stop script 'config'
shutdown: [pid 91515]
Shutdown NOW!

*** FINAL System shutdown message from root@xxx ***

System going down IMMEDIATELY

mmetc · 2024-10-08T08:49:27Z

I would like to share my testing related to the original issue.
I have found the below during testing: -
[...]

Thanks for testing! You can update the script with this command until 1.6.3-2 is out.
# fetch -o /usr/local/etc/rc.d/crowdsec https://github.com/crowdsecurity/plugins/releases/download/crowdsec-1.6.3-2-hotfix/crowdsec
Thanks @mmetc - Great news, initial testing is showing that this fix has corrected the issue. Any chance you can have a look at the other issue with Crowdsec? Stop/Start button issue - #4280.

Yes, I expect the start/stop button to work after the fetch. If it doesn't, I can inspect the crowdsec logs for any other issue. Run "cscli support dump" and send the file to support@crowdsec.net (partial configuration is included, without password/api keys of course)

daygle · 2024-10-08T08:54:45Z

Thanks again @mmetc.

Just letting you know that the 'Play' button remains green after CrowdSec is stopped - still an issue I am afraid. I'll send the logs to that email address.

mmetc · 2024-10-08T10:09:27Z

Seen and replied. As I wrote in the other issue, run "killall crowdsec" after the fetch to make sure there's no orphan process.

RudiKlein · 2024-10-08T16:28:03Z

@mmetc I have reproduced the issue I still have after installing the hotfix.

Hotfix installed
$ killall crowdsec didn't kill the crowdsec processes.
$ reboot to the rescue.

Status just after boot

0 28250     1 0  68  0   12732    2272 kqread   Is    -    0:00.00 daemon: crowdsec[28405] (daemon)
0 28405 28250 0  63  0 1482580  162092 kqread   I     -    0:32.90 /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
0 37953 37094 1  20  0 1239152   44728 uwait    S     -    0:01.35 /usr/local/bin/crowdsec-firewall-bouncer -c /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml -v (crowdsec-firewall-b)

Status after STOP from Services page (won't go to STOPPED state.)

0  1208   263 1  20  0   92144   64120 select   S     -    0:00.73 /usr/local/bin/php /usr/local/sbin/pluginctl -s crowdsec stop
0  1986   263 1  68  0   13280    3056 wait     I     -    0:00.01 /bin/sh /usr/local/etc/rc.d/oscrowdsec stop
0  2520  1986 1  68  0   13280    3172 wait     I     -    0:00.01 /bin/sh /usr/local/etc/rc.d/crowdsec stop

Status after $ killall crowdsec

0  1986   263 1  68  0   13280    3056 wait     I     -    0:00.01 /bin/sh /usr/local/etc/rc.d/oscrowdsec stop
0  2520  1986 1  68  0   13280    3172 wait     I     -    0:00.01 /bin/sh /usr/local/etc/rc.d/crowdsec stop
0 28405     1 2  20  0 1482580  163008 uwait    S     -    0:36.23 /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml

I then used $ killall -9 crowdsec to kill the crowdsec processes succesfully.
After that stopping/starting from the Services page worked fine.

It seems there's still an issue after the hotfix.

Rudi

daygle · 2024-10-08T18:44:21Z

@RudiKlein It does awfully sound the same as what I was experiencing before the hotfix.

These are the steps I followed:-

Stopped Crowdsec via the services 'Stop' button.
Applied fix.
Rebooted OPNsense.
Stopped Crowdsec via the services 'Stop' button.
Killed Crowdsec via shell command.
Changed Crowdsec port from 8080 to 8085.
Started Crowdsec via the services 'Play' button.

I then was able to stop/restart Crowdsec via services and perform reboots without any issues so far.

mmetc · 2024-10-09T07:14:22Z

@mmetc I have reproduced the issue I still have after installing the hotfix.

I then used $ killall -9 crowdsec to kill the crowdsec processes succesfully. After that stopping/starting from the Services page worked fine.

It seems there's still an issue after the hotfix.

I changed the recommendation to "killall -9", that should do it, thanks! And 24.7.6 is coming with the proper package.

LaurenceJJones · 2024-10-09T14:14:36Z

To keep thread updated 24.7.6 has just been released which includes the proper package @mmetc described

refs:
https://discourse.crowdsec.net/t/bug-opnsense-24-7-5-crowdsec-1-6-3/2057
https://forum.opnsense.org/index.php?topic=43305.0

fichtner · 2024-10-09T14:41:44Z

@mmetc @LaurenceJJones thanks guys, closing this then :)

fichtner · 2024-10-09T14:52:16Z

(if that was too soon we can reopen. getting mixed signals on reddit.)

LaurenceJJones · 2024-10-09T14:53:42Z

(if that was too soon we can reopen. getting mixed signals on reddit.)

I believe the issue is some processes are still not getting killed cause the patch is only taking effect once the currently running PID is killed hence why in our thread we mention there has to be manual intervention. Once the original PID is killed the new patch takes effect and should have no issues.

If you are pre patch run these commands:

fetch -o /usr/local/etc/rc.d/crowdsec https://github.com/crowdsecurity/plugins/releases/download/crowdsec-1.6.3-2-hotfix/crowdsec
killall -9 crowdsec

then update your opnsense deployment

If you are stuck mid patch and waiting for the pid to be killed then run just

killall -9 crowdsec

This will allow opnsense to update and reboot as per normal. (the previous fetch contents are included in the CrowdSec update package so no need to run it again)

fichtner · 2024-10-09T14:59:25Z

Judging purely by opnsense/ports@4bd513d58 which gets installed prior to reboot it should fix the impending reboot as the updated script is called to stop it. That is, of course, under the assumption the fix is accurate.

Cheers,
Franco

fichtner · 2024-10-09T15:08:18Z

Ok so the issue is users having attempted the 24.7.5 update having the process stuck and will then continue to be stuck if attempting to go to 24.7.6 directly from the bad 24.7.5 state. But any forced reboot or the mentioned workaround would make it unstuck. Understood now.

GitTimeraider · 2024-10-09T15:16:04Z

Hmm.. had it last updates and have it again now. Kinda annoying having to manually intervene each time.

LaurenceJJones · 2024-10-09T15:20:19Z

Hmm.. had it last updates and have it again now. Kinda annoying having to manually intervene each time.

Yes this patch fixes it so you will not have to manually intervene moving forward but to do it you need to do it one more time 😓

LowHanger · 2024-10-09T15:30:34Z

For what it's worth, I just had the same issue when upgrading from 24.5_3 to 24.6. I ssh'ed and killed the PID which was stuck. When I did that the install process on the GUI continued on as normal.

julsssark · 2024-10-09T17:02:23Z

Thank you for fixing this!

roens · 2024-10-10T07:00:24Z

Nice work folks!

I just ran an upgrade from 24.7.5_3 to 24.7.6.

Of course it did hang on waiting for Crowdsec to quit, and I had to killall -9 crowdsec. It did then proceed with the remainder of the upgrade, then rebooted.

Looking forward to future unfettered reboots.

daygle · 2024-10-10T10:41:38Z

@mmetc - Some not so good news I am afraid.

After upgrading to OPNsense 24.7.6 and rebooting a number of times the same issue is happening :(

mmetc · 2024-10-10T10:56:19Z

@daygle can you please run "cscli support dump" and send the output to support@crowdsec.net? Thanks

daygle · 2024-10-10T11:04:42Z

@daygle can you please run "cscli support dump" and send the output to support@crowdsec.net? Thanks

Thanks @mmetc - sent you an email and support logs.

benisdev-py · 2024-11-08T15:06:15Z

same problem still every update...

julsssark · 2024-11-08T15:08:36Z

I upgraded to 24.7.8 from 24.7.7 and the update worked without a problem.

benisdev-py · 2024-11-08T15:11:01Z

Oh okay did you have installed also crowdsec firewall bouncer?

julsssark · 2024-11-08T15:13:11Z

Yes. Crowdsec was installed and bouncer was running when I did the update. What version of opnsense were you upgrading from/to? You need to have been running at least 24.7.6 for the fix to be active (i.e., if you are upgrading from a version before 24.7.6 to a version => 24.7.6, you will still get a hang on the first update). Subsequent upgrades should work just fine.

benisdev-py · 2024-11-08T16:18:48Z

I also updater from 24.7.6 to a version => 24.7.6

fichtner added support Community support upstream Third party issue and removed support Community support labels Oct 1, 2024

fichtner mentioned this issue Oct 1, 2024

No automatic reboot after upgrade? opnsense/core#7927

Closed

2 tasks

fichtner closed this as completed Oct 9, 2024

fichtner assigned mmetc Oct 9, 2024

Reboot held up by Crowdsec refusing to stop #4262

Reboot held up by Crowdsec refusing to stop #4262

Comments

roens commented Sep 28, 2024 • edited Loading

Starkstromkonsument commented Sep 28, 2024

fichtner commented Sep 28, 2024

davesnthere commented Oct 1, 2024 • edited Loading

kumy commented Oct 1, 2024 • edited Loading

fichtner commented Oct 1, 2024

RudiKlein commented Oct 2, 2024

fichtner commented Oct 2, 2024

Unspec7 commented Oct 2, 2024 • edited Loading

nhatlinh1982 commented Oct 4, 2024

daygle commented Oct 5, 2024 • edited Loading

mmetc commented Oct 7, 2024 • edited Loading

mmetc commented Oct 7, 2024

RudiKlein commented Oct 7, 2024

daygle commented Oct 7, 2024 • edited Loading

nhatlinh1982 commented Oct 8, 2024 • edited Loading

daygle commented Oct 8, 2024

mmetc commented Oct 8, 2024 • edited Loading

daygle commented Oct 8, 2024

mmetc commented Oct 8, 2024

daygle commented Oct 8, 2024

mmetc commented Oct 8, 2024

RudiKlein commented Oct 8, 2024

daygle commented Oct 8, 2024

mmetc commented Oct 9, 2024 • edited Loading

LaurenceJJones commented Oct 9, 2024

fichtner commented Oct 9, 2024

fichtner commented Oct 9, 2024

LaurenceJJones commented Oct 9, 2024 • edited Loading

fichtner commented Oct 9, 2024

fichtner commented Oct 9, 2024

GitTimeraider commented Oct 9, 2024

LaurenceJJones commented Oct 9, 2024 • edited Loading

LowHanger commented Oct 9, 2024

julsssark commented Oct 9, 2024

roens commented Oct 10, 2024 • edited Loading

daygle commented Oct 10, 2024 • edited Loading

mmetc commented Oct 10, 2024

daygle commented Oct 10, 2024

benisdev-py commented Nov 8, 2024

julsssark commented Nov 8, 2024 • edited Loading

benisdev-py commented Nov 8, 2024

julsssark commented Nov 8, 2024 • edited Loading

benisdev-py commented Nov 8, 2024

roens commented Sep 28, 2024 •

edited

Loading

davesnthere commented Oct 1, 2024 •

edited

Loading

kumy commented Oct 1, 2024 •

edited

Loading

Unspec7 commented Oct 2, 2024 •

edited

Loading

daygle commented Oct 5, 2024 •

edited

Loading

mmetc commented Oct 7, 2024 •

edited

Loading

daygle commented Oct 7, 2024 •

edited

Loading

nhatlinh1982 commented Oct 8, 2024 •

edited

Loading

mmetc commented Oct 8, 2024 •

edited

Loading

mmetc commented Oct 9, 2024 •

edited

Loading

LaurenceJJones commented Oct 9, 2024 •

edited

Loading

LaurenceJJones commented Oct 9, 2024 •

edited

Loading

roens commented Oct 10, 2024 •

edited

Loading

daygle commented Oct 10, 2024 •

edited

Loading

julsssark commented Nov 8, 2024 •

edited

Loading

julsssark commented Nov 8, 2024 •

edited

Loading