Falco does not properly start after upgrade #400
Labels
Comments
|
Had the same thing happen now with the 0.12.1 version of Falco |
|
I wonder if this is the same cause as #418. The transition from 0.11.1 to 0.12 included an incompatible kernel module change, so it's possible that falco crashed after the new install because it was trying to use the already-loaded old kernel module. I know this was a while back, but are you still experiencing this problem and if so could you check /var/log/messages to see if falco had a segfault? |
|
A bit too far for the log archives I'm afraid as the messages.log have been rotated and I can't find anything older than early October. I guess I will see with version 0.12.2 or 0.13.0 (whichever is first) 🙂 |
|
/usr/lib/dkms/dkms_autoinstaller start |
|
Still happening now with the upgrade to 0.13.0. Tried running the |
|
did you confirm the old kernel module is indeed removed and the new version loaded? |
|
@mfdii The un-upgraded servers do appear to be running the 0.12.1 version, if I am not entirely mistaken. The contents of While on one of the upgraded servers it is: I might also add that no sefaults were spotted in /var/log/messages during the upgrade. |
|
Output from terminal when upgrading one of the servers to 0.13.0 tonight: As you also can see from the beginning there, the module version was 0.12.1 before upgrading.
Then a minut pause until I manually start it with |
|
If you do a `dmesg | grep falco` do you see the 0.13.0 kernel module being loaded? If not do a `rmmod falco-probe && modprobe falco-probe`
… On Nov 13, 2018, at 12:39 PM, Alexander Karlstad ***@***.***> wrote:
Output from terminal when upgrading one of the servers to 0.13.0 tonight:
21:33 ***@***.*** /tmp]# apt install falco
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
falco
1 upgraded, 0 newly installed, 0 to remove and 43 not upgraded.
Need to get 0 B/3 125 kB of archives.
After this operation, 503 kB of additional disk space will be used.
Reading changelogs... Done
(Reading database ... 115797 files and directories currently installed.)
Preparing to unpack .../falco_0.13.0_amd64.deb ...
-------- Uninstall Beginning --------
Module: falco
Version: 0.12.1
Kernel: 4.9.0-5-amd64 (x86_64)
-------------------------------------
Status: Before uninstall, this module version was ACTIVE on this kernel.
falco-probe.ko:
- Uninstallation
- Deleting from: /lib/modules/4.9.0-5-amd64/updates/dkms/
- Original module
- No original module was found for this module on this kernel.
- Use the dkms install command to reinstall any previous module version.
depmod....
DKMS: uninstall completed.
-------- Uninstall Beginning --------
Module: falco
Version: 0.12.1
Kernel: 4.9.0-8-amd64 (x86_64)
-------------------------------------
Status: Before uninstall, this module version was ACTIVE on this kernel.
falco-probe.ko:
- Uninstallation
- Deleting from: /lib/modules/4.9.0-8-amd64/updates/dkms/
- Original module
- No original module was found for this module on this kernel.
- Use the dkms install command to reinstall any previous module version.
depmod....
DKMS: uninstall completed.
------------------------------
Deleting module version: 0.12.1
completely from the DKMS tree.
------------------------------
Done.
Unpacking falco (0.13.0) over (0.12.1) ...
Setting up falco (0.13.0) ...
Installing new version of config file /etc/falco/falco_rules.yaml ...
Installing new version of config file /etc/falco/rules.available/application_rules.yaml ...
Installing new version of config file /etc/falco/falco_rules.local.yaml ...
Loading new falco-0.13.0 DKMS files...
Building for 4.9.0-8-amd64
Building initial module for 4.9.0-8-amd64
Done.
falco-probe:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/4.9.0-8-amd64/updates/dkms/
depmod...
DKMS: install completed.
Processing triggers for systemd (232-25+deb9u4) ...
As you also can see from the beginning there, the module version was 0.12.1 before upgrading.
/var/log/messages after/during the upgrade:
Nov 13 21:33:46 server kernel: [2672998.045889] falco_probe: deallocating consumer ffff96777ff93000
Nov 13 21:33:46 server kernel: [2672998.071596] falco_probe: no more consumers, stopping capture
Then a minut pause until I manually start it with service falco start:
Nov 13 21:34:53 server falco: Falco initialized with configuration file /etc/falco/falco.yaml
Nov 13 21:34:53 server falco: Loading rules from file /etc/falco/falco_rules.yaml:
Nov 13 21:34:53 server falco: Loading rules from file /etc/falco/falco_rules.local.yaml:
Nov 13 21:34:53 server falco: Loading rules from file /etc/falco/rules.d/local.yaml:
Nov 13 21:34:54 server kernel: [2673065.527547] falco_probe: adding new consumer ffff9677889af100
Nov 13 21:34:54 server kernel: [2673065.527598] falco_probe: initializing ring buffer for CPU 0
Nov 13 21:34:54 server kernel: [2673065.535927] falco_probe: CPU buffer initialized, size=8388608
Nov 13 21:34:54 server kernel: [2673065.535928] falco_probe: initializing ring buffer for CPU 1
Nov 13 21:34:54 server kernel: [2673065.544242] falco_probe: CPU buffer initialized, size=8388608
Nov 13 21:34:54 server kernel: [2673065.544243] falco_probe: initializing ring buffer for CPU 2
Nov 13 21:34:54 server kernel: [2673065.552423] falco_probe: CPU buffer initialized, size=8388608
Nov 13 21:34:54 server kernel: [2673065.552425] falco_probe: initializing ring buffer for CPU 3
Nov 13 21:34:54 server kernel: [2673065.560346] falco_probe: CPU buffer initialized, size=8388608
Nov 13 21:34:54 server kernel: [2673065.560348] falco_probe: initializing ring buffer for CPU 4
Nov 13 21:34:54 server kernel: [2673065.568472] falco_probe: CPU buffer initialized, size=8388608
Nov 13 21:34:54 server kernel: [2673065.568473] falco_probe: initializing ring buffer for CPU 5
Nov 13 21:34:54 server kernel: [2673065.575993] falco_probe: CPU buffer initialized, size=8388608
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
I can't say there are any specific messages regarding loading/unloading the kernel module in dmesg no: Looking back in dmesg to October 14 I do see a
|
|
For what it's worth, this seems to also happen when upgrading to 0.13.1. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
This is still happening though, in Debian. When I have also noticed that |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Using apt, when upgrading Falco, the last few versions does not start up again after a completed upgrade. The service remains stopped.
I get a Slack notification when the service is stopped and started (
sudo service falco start/stop), but when upgrading, I am only notified of it being stopped. When logging on the server later to check, it is indeed still stopped.Manually running
sudo service falco startstarts the service back up without issues.To be fair I thought maybe this was handled automatically by systemd, but I only have this problem with Falco at the moment.
Servers are running Debian 9.5.
The text was updated successfully, but these errors were encountered: