Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Kernel panic in Ubuntu 14.04 #3

Open
hotice opened this Issue · 123 comments

7 participants

@hotice

douane-dkms causes Kernel panics in Ubuntu 14.04 64bit. This seems to occur randomly, while browsing some websites in Firefox for instance (so not on boot).

I'm using the Ubuntu Linux 3.13.0-24-generic Kernel.

@zedtux
Owner

Do you know the amount of tabs that was opened at that moment ?

I mean you got the freeze while browsing 100 pages (:smile:) or only 1 page ?

@hotice

There were a few tabs, I don't know how many...

But it happened twice: once I got the kernel panic message and once the system just froze and nothing else happened. After removing douane (about 2-3 hours ago), that didn't happen any more.

I'll continue to use my computer without douane today and if there aren't any more freezes, I'll install douane again tomorrow and use it for the whole day to see if this is indeed because of douane.

So for now... this isn't confirmed :)

@zedtux zedtux referenced this issue in Douane/Douane
Open

Discussion about Douane #27

@zedtux
Owner

I think you're going to have the freeze ... :-(

Now we need to identify the issue. The first thing you should do is to compile the kernel module with the debug information (could be a good idea to have debug enabled package ;-)).
To do so, you have to add the compilation flag -DDEBUG. You can have a look here.

This will print a lot more of debugs in the /var/log/kern.log file. When a kernel panic occurred and you've rebooted, then look at the kern.log file and upload it in this ticket.
(You should reset the file before to enable douane until the freeze).
Also if you could mention what you was doing at the moment of the freeze would be great (Did you had torrents download, how many tabs in your browser, did you had the update manager running, ...).

I will write a wiki with a procedure to follow when having a freeze so that next time someone have an issue like this, I will redirect him to the wiki.

@zedtux
Owner

I have merge the PR #2 which could solve your issue. Can you use the version from master ?

@hotice

Indeed, I tried it yesterday and today and got a freeze and a kernel panic with Douane running and no issues when Douane wasn't running. However, this time I didn't uninstall it, I just disabled the daemon and the crashes stopped occurring. I'll try the latest master.

@zedtux
Owner

Alright, I'm looking for your comments.

@zedtux
Owner

BTW I'm using the kernel version 3.13.0-24-generic.

@hotice

And... it happened again with the latest master. Here's the log: https://dl.dropboxusercontent.com/u/1113424/kern.log

But I don't know if I enabled debugging properly because for the deb I use a different makefile.

When it happened, Firefox was open but I wasn't using it, I was uploading a package to a PPA via dput. So I was working in a terminal.

@zedtux
Owner

@pavlinux is doing an amazing work on the kernel module. He found 2 other memoryleaks. Can you please test with the new master ?

Regarding the debug, you didn't enabled it.
Normally in your package you're passing the compilation flag -g -DDOUANE_VERSION=\"$(MODULE_VERSION)\" and you need to add -DDEBUG.
I'm going to check your packaging debian files and try to help you on that.

@hotice

OK, I'll try the latest master. Here are the makefiles I use in the deb: https://dl.dropboxusercontent.com/u/1113424/makefiles.tar.xz

@hotice

And it happened again... I had just rebooted, started Firefox, loaded 5-6 tabs and the Kernel panic occured. It also completely fucked up my Firefox profile.

@zedtux
Owner

Regarding the Makefile, there are those lines:

ifdef DEBUG
  CFLAGS_$(obj-m) := -DDEBUG
endif

They looks good but when do you set the DEBUG ?
And here you're missing to pass the version from the VERSION file.
If you execute modinfo douane in a terminal you should have the version shown. If you don't pass the -DDOUANE_VERSION compilation flag, the version should be 'UNKNOWN'.

Now regarding your freezes it's really strange. We are using the same kernel version, we are both 64 bits. I don't see what could be different that freeze your machine and not mine ...

Try compiling the kernel module with the debug enabled and then send me the kern.log as you did before otherwise I can't help you.

And I'm really sorry for your Firefox profile ... :-((

@zedtux zedtux added the bug label
@hotice

I don't know how to set the debug, just removing "ifdef DEBUG" should be enough? As for version, it's set in Makefile.dkms but indeed, "modinfo douane" doesn't return the version. How do I fix that?

@zedtux
Owner

For now, in order to solve the freezes, yes just remove the ifdef DEBUG line in order to enable it.
Then when it will be fixed, if you agree, you should produce 2 packages:

  • douane-dkms which wouldn't have the -DDEBUG flag
  • douane-dkms-dbg which would have the -DDEBUG flag

Regarding the daemon version, in the Makefile you have to add the compilation flag -DDOUANE_VERSION=\"$(MODULE_VERSION)\" where MODULE_VERSION is the content of the VERSION file.

@hotice

With debugging enabled I got so many kernel panics, I don't even know where to begin.

So, I installed douane-dkms with debugging enabled, tried to restart my laptop -> kernel panic. Tried to start the laptop -> kernel panic. After another reboot I managed to log in, cleared kern.log, tried to run "sudo apt-get update" -> segflault for apt when trying to get some https so I figured this is related to secure connections. Started Chromium and before it could even start -> kernel panic.

Rebooted -> kernel panic on startup. Rebooted > disabled daemon from tty and finally got the kern.log file so here it is: https://dl.dropboxusercontent.com/u/1113424/kern2.log

@zedtux
Owner

Question: Have you uninstalled the old douane from months when you was trying douane ?
I'm wondering if there's any chances that you're using an old version of the kernel module.

Could you try to uninstall everything regarding Douane, then ensure that you no more have the kernel module (any version) with dkms:

$ dkms status

On my computer this is the output:

douane-testing, 0.8.2-trusty1, 3.13.0-23-generic, x86_64: installed
douane-testing, 0.8.2-trusty1, 3.13.0-24-generic, x86_64: installed
virtualbox, 4.3.10, 3.13.0-23-generic, x86_64: installed

As you can see here, I have 2 different versions but for 2 different kernels, and then virtualbox.

@hotice

Everything related to Douane was purged the first time I got the kernel panic.

$ dkms status
bbswitch, 0.7, 3.13.0-24-generic, x86_64: installed
douane, 0.8.2, 3.13.0-24-generic, x86_64: installed
nvidia-331, 331.38, 3.13.0-24-generic, x86_64: installedError! Could not locate dkms.conf file.
File: does not exist.
v4l2loopback, 0.8.0, 3.13.0-24-generic, x86_64: installed```

(I fixed the version)
$ modinfo douane
filename: /lib/modules/3.13.0-24-generic/updates/dkms/douane.ko
license: GPL
version: 0.8.2
author: Guillaume Hain zedtux@zedroot.org
description: Douane
srcversion: 89A06BC09BFD7C6200B63A2
depends:

vermagic: 3.13.0-24-generic SMP mod_unload modversions

@zedtux
Owner

OK, so we are sure you're using the last version.

Your kern.log file is interesting. I'm looking at the issue and let you know.

@hotice

Yes, and the log shows that douane is causing the kernel panic, considering that the modules listed when the panic occurred are iptables...:

May 1 13:40:41 ubuntu-desktop kernel: [ 24.595858] douane:166:clear_rules: Rules successfully cleaned.
May 1 13:40:41 ubuntu-desktop kernel: [ 24.595903] ------------[ cut here ]------------
May 1 13:40:41 ubuntu-desktop kernel: [ 24.595962] kernel BUG at /build/buildd/linux-3.13.0/mm/slub.c:3365!
May 1 13:40:41 ubuntu-desktop kernel: [ 24.596039] invalid opcode: 0000 [#1] SMP
May 1 13:40:41 ubuntu-desktop kernel: [ 24.596111] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc nvram ctr ccm v4l2loopback(OF) pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables douane(OF) bbswitch(OF) bnep rfcomm dm_crypt uvcvideo binfmt_misc videobuf2_vmalloc videobuf2_memops videobuf2_core videodev btusb bluetooth joydev dell_wmi sparse_keymap dell_laptop dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse serio_raw snd_hda_codec_hdmi arc4 snd_hda_codec_realtek iwldvm snd_hda_intel snd_hda_codec mac80211 snd_hwdep snd_pcm snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer iwlwifi cfg80211 lpc_ich snd mei_me mei soundcore wmi mac_hid parport_pc ppdev coretemp lp parport hid_logitech_dj hid_generic usbhid

@hotice

By the way, this might also be related to Douane and VirtualBox, try installing VirtualBox and see if that gets this reproducible....

@zedtux
Owner

Well I have Virtualbox as you can see in one of my comment, but is not running.
I will try to use it and I let you know.

Another question: Have you douane installed on your host OS or in a guest OS in Virtualbox ?

@hotice

On my OS, not in Virtualbox. I was talking about VirtualBox because I though its network modules might in some way be affecting this, but if you have it installed too, that's not related so I was wrong.

@zedtux
Owner

I have removed the call causing the kernel panic shown in your kern.log.
Can you please try again with the new code from master and update your kern.log for the next kernel panic.

(To be more clear, the kernel panic you've encountered relates to the last pull request in my opinion (I'm investigating in order to understand it). Now I have reverted the code so that you're going back to the previous kernel panic.
Now that you have the debug mode enabled I can see what's wrong and fix it.)

@hotice

Kernel panic again, here's the log: https://dl.dropboxusercontent.com/u/1113424/kern3.log

Also, this time I knew exactly how to reproduce it and just as I suspected, it worked: open Chromium, open GitHub -> kernel panic. That's weird :) (I'm not trying it with Firefox any more, I can't have another profile messed up :D).

@zedtux
Owner

At least this is cool that you can reproduce ! :+1:

Let us know if the @pavlinux branch works or not. I have another idea if it's not working.

@hotice

It happened again with @pavlinux 's change too. Here's the log: https://dl.dropboxusercontent.com/u/1113424/kern4.log

I once again reproduced it by opening Chromium, but I didn't have to load GitHub: when opening Chromium, no page would load (it restored 6 tabs, none would load) and I waited a bit and then I got a kernel panic.

@zedtux
Owner

@hotice please try with the new master.

@pavlinux

This is bad, that the memory is freed implicitly somewhere.
After allocation only one function may do it
nlmsg_put(skb,...);

@pavlinux

Need replace
skb = alloc_skb(NLMSG_SPACE(sizeof(struct network_activity)), GFP_ATOMIC);
by
skb = nlmsg_new(NLMSG_ALIGN(sizeof(struct network_activity)) + nla_total_size(1), GFP_KERNEL);

@hotice

I'll wait until that's fixed then.

@pavlinux

I have a sign - GFP_ATOMIC only use when working with devices. :)

@zedtux
Owner

It's in master, and the build pass on kernel 3.13.

@hotice

This time my desktop crashed as soon as I installed the new dkms. Also, I was unable to reach my desktop after a reboot (well, I tried about 3 reboots) so I had to purge everything via tty so I could get back to my desktop. So this is getting worse and worse... Here's the log: https://dl.dropboxusercontent.com/u/1113424/kern5.log

@zedtux
Owner

@hotice I was wondering if your packages in your PPA have the latest sources ?
If it's the case, especially for douane-dkms, I could try them on my machine.

@hotice

It's the latest sources from yesterday. I'll have to update douane-dkms and since the code changed I'll have to bump the package version to 0.8.3. It will be in the PPA in a few minutes. The PPA is here: https://launchpad.net/~nilarimogard/+archive/test4

@hotice

I uploaded the latest dkms, but like I said it's now version 0.8.3 (because of my makefile patches, I can't reupload the same .orig.tar.gz on launchpad if the contents are different so I had to bump the version). The package should be ready in about 15 minutes or so, you can track it here: https://launchpad.net/~nilarimogard/+archive/test4/+packages

@hotice

One more thing: the dkms packages are built with debugging enabled!

@zedtux
Owner

Thank you @hotice.

@zedtux
Owner

OK I'm now testing your packages.

@zedtux
Owner

That's wired but the installation of douane meta-package didn't installed the libboost-* ...

@zedtux
Owner

The good news is that I have the same kernel panic than you !
Well... with the version I had before, I had no kernel panic ... I'm going to try to revert the code as of before and see if it's better.

@zedtux
Owner

First thing I have found:

After having installed the packages I had 2 running douane kernel module !! I don't how is this possible but executing sudo dkms status was showing me 1 line for douane, and after having removed the kernel module manually, running sudo dkms status shows nothing, but I still have new lines of log in the /var/log/douane.log file !

@zedtux
Owner

Now when I remove and reinstall the package there no running kernel module ... there is something wrong in the package I think. I'm trying to figure out what...

@pavlinux

Guys.... my mistake - after netlink_unicast() not need to free()

@zedtux
Owner

Here is the kernel panic extract:

May  1 20:25:13 zUbuntu kernel: [ 1111.196530] Call Trace:
May  1 20:25:13 zUbuntu kernel: [ 1111.196537]  [<ffffffff8160bb7e>] skb_free_head+0x1e/0x80
May  1 20:25:13 zUbuntu kernel: [ 1111.196542]  [<ffffffff8160bcb6>] skb_release_data+0xd6/0x110
May  1 20:25:13 zUbuntu kernel: [ 1111.196548]  [<ffffffffa04a35f7>] ? netfiler_packet_hook+0x9c7/0xdd0 [douane]
May  1 20:25:13 zUbuntu kernel: [ 1111.196554]  [<ffffffff8160bd14>] skb_release_all+0x24/0x30
May  1 20:25:13 zUbuntu kernel: [ 1111.196559]  [<ffffffff8160bd72>] kfree_skb+0x32/0x90
May  1 20:25:13 zUbuntu kernel: [ 1111.196564]  [<ffffffffa04a35f7>] netfiler_packet_hook+0x9c7/0xdd0 [douane]
May  1 20:25:13 zUbuntu kernel: [ 1111.196571]  [<ffffffffa0225a80>] ? get_unique_tuple+0x280/0x660 [nf_nat]
May  1 20:25:13 zUbuntu kernel: [ 1111.196578]  [<ffffffff81653b20>] ? ip_forward_options+0x1c0/0x1c0
May  1 20:25:13 zUbuntu kernel: [ 1111.196584]  [<ffffffff81649e8a>] nf_iterate+0x9a/0xb0
May  1 20:25:13 zUbuntu kernel: [ 1111.196589]  [<ffffffff81653b20>] ? ip_forward_options+0x1c0/0x1c0
May  1 20:25:13 zUbuntu kernel: [ 1111.196594]  [<ffffffff81649f14>] nf_hook_slow+0x74/0x130
May  1 20:25:13 zUbuntu kernel: [ 1111.196599]  [<ffffffff81653b20>] ? ip_forward_options+0x1c0/0x1c0
May  1 20:25:13 zUbuntu kernel: [ 1111.196604]  [<ffffffff816559f2>] __ip_local_out+0xa2/0xb0

The wired thing is that it looks like the douane kernel module is called twice:

  • First time line 7, it does a call to kfree_skb and it looks like during the call to skb_release_all
  • Second time line 4, the kernel module is called again, but this time the name of the douane method starts with a question mark and maybe try to free something NULL ...

What do you think @pavlinux ?

@zedtux
Owner

Well I'm now running the module from the commit https://github.com/Douane/douane-dkms/tree/0f37fe644feb26ae6d60fa03520cd640d75727a1 and no more kernel panic until now.

@zedtux
Owner

I had kernel panics only when running apt-get update... but it was more stable.

I'm going to try your fork @pavlinux.

@zedtux
Owner

@hotice you didn't changed the module version in the dkms.conf file.

@zedtux
Owner

I just realized that I have forgotten to do dkms build so I have installed again the module from your package @hotice.

So I did the following:

sudo rmmod douane
sudo dkms uninstall -m douane -v 0.8.3
sudo dkms remove -m douane -v 0.8.3 --all
cd /usr/src/douane-0.8.3/
sudo mv douane.c douane.c.orig
sudo wget https://raw.githubusercontent.com/Douane/douane-dkms/903c26a621d84308675e4ec249bee46f2d04eb42/douane.c
sudo dkms build -m douane -v 0.8.3
sudo dkms install -m douane -v 0.8.3
sudo modprobe douane

Then in order to ensure that there's no kernel panic, I've opened 2 terminals with the following commands:

  • tail -f /var/log/kern.log
  • tail -f /var/log/kern.log | grep "kernel BUG at"

The last command will show if a kernel bug occur somewhere.
@hotice can you please try and let me know ?

@zedtux
Owner

Now since half an hour, with Birdie, firefox, apt-get and no kernel bug/panic. :)

@hotice

Is the fix now in git here or do I still have to replace that file manually?

@zedtux
Owner

I'm waiting your validation before to revert the code in master.
In the other hand I have created a tag 0.8.2 which is the 'fixed' version if you validate.

@hotice

@zedtux Hmmm your commands are wrong. You move douane.c to douane.c.orig twice. And then you move douane.c.1 to douane.c but douane.c.1 doesn't exist. Please fix them.

@zedtux
Owner

Mmmm sorry my bad... I have updated my commands.

@hotice

Well, opening GitHub in Chromium didn't result in any kernel panics. So far so good.

@pavlinux

This is in no way associated with latest fixes :)

@hotice

Well, there were no more kernel panics / desktop freezes. So whatever caused it doesn't happen any more with that commit.

@zedtux
Owner

Great news ! So could you please build a package with the 0.8.2 tag ? (Maybe you should create a new PPA ... sorry for this.
(The 0.8.2 version is frozen so there will be no more commit for this tag).

@zedtux
Owner

Sorry, can we consider to close this issue ?

@hotice

Yes, I'll create a new PPA. Closing this.

@hotice hotice closed this
@pavlinux

@hotice You have a computer, sorry, zoo: Virtualbox, samba, Apparmor, systemd...

So, I think panics still appear

@zedtux
Owner

@pavlinux are you using your fork on your machine ?

If yes, which are the services you're using ?

@hotice

@pavlinux I maintain a blog (www.webupd8.org) and hundreds of PPA packages and I test a lot of things for both so I have to have lot of things installed.

@hotice

And besides VirtualBox, you mentioned standard stuff that's installed by default in Ubuntu... samba, apparmor, systemd (well, Ubuntu didn't switch to it yet but there are parts of it implemented already).

@pavlinux

@zedtux Debian 7, kernels: 3.2.57, 3.14.2, Not constantly running services, only system: alsa, cron, dbus, syslog, udev, acpid. No SeLinux, apparmor, grsecirity, etc. ... I do not have paranoid delusions of persecution :)

When testing applications, then work: MySQL, Samba, DHCP, Bind, tftp, Postgres, Vmware, stunnel, openvpn, qemu,... NTP, XBT torrrent tracker, torrent client

@hotice

@zedtux I built a new PPA and I'm not using the packages from there. So far so good. Here's the PPA: https://launchpad.net/~nilarimogard/+archive/test2/ - can you please use it too and see if everything is ok? (douane-dkms is no longer built with debugging enabled; I built douane-dkms-dbg for that but don't install it for now, I have to change some dependencies for some other packages to get it to install properly).

@zedtux
Owner

OK I will install them this evening when I will be home and let you know.

@hotice

I had another kernel panic so it's not fixed. I didn't have debugging enabled unfortunately...

@zedtux
Owner

:-(

Then can you please install your -dgb package (and I will do the same) ?

@hotice

I installed it now. But now all I can do is wait, it happened after a few hours the last time...

@hotice

Hmmm my douane-dkms-dbg package doesn't work. Here's a douane-dkms deb with debugging enabled: https://dl.dropboxusercontent.com/u/1113424/douane-dkms_0.8.2-1%7Ewebupd8%7Etrusty0_amd64.deb

@zedtux
Owner

Of course ... Then we will see.

@zedtux zedtux reopened this
@zedtux
Owner

OK So I have installed your package @hotice. Let's see.

Do you have news on your side ?
Can't you try to grep in your kern.log file ?

@zedtux
Owner

Here is my modinfo:

modinfo douane
filename:       /lib/modules/3.13.0-24-generic/updates/dkms/douane.ko
license:        GPL
version:        0.8.2
author:         Guillaume Hain <zedtux@zedroot.org>
description:    Douane
srcversion:     A8F1BF0FD4C46D373F924F5
depends:        
vermagic:       3.13.0-24-generic SMP mod_unload modversions

Can you please post yours ?

@zedtux
Owner

Just few remarks for your packaging:

  • The license should be 'GPL v2'. You didn't took the correct version. I guess the issue is LP which can't checkout a tag ?
  • You should update the prerm and postinst script in order to stop and start the daemon

Edit: Sorry ... I have double checked for the license, and the license is wrong in the tag ... I'm going to change it.

Edit 2: I can't update the license in the tag. Can you checkout a specific branch in LP ? If yes, then I could create a branch per versions instead of a tag.

@zedtux
Owner

It is now about hours I'm using the package from your link and no kernel panic/bug/freeze.
I have running since the hours Spotify (so a lot of activities non-stop) and I'm using Firefox. I'm going now to also open Birdie... but I'm not sure to have any freezes...

@zedtux
Owner

Finally, some hours later I got a kernel panic, but unfortunately it wasn't saved in the kern.log file ... :-(

@zedtux
Owner

Yes, I'm going to do it now.

@zedtux
Owner

I got 2 freezes in less than 15 minutes (with Spotify, Firefox opened with jolicould.com and birdie) where I don't see any kernel panic in the logs (while with the other version we see a kernel panic).

So the compute freezes and you can't do anything else than a reset.

@zedtux
Owner

I'm now checking the kernel module with the KEDR framework.
I'm copying a file of 4.7GB from my NAS to my localhost.

Without the daemon enable, I have the following output (and no freeze):

May  3 14:12:34 zUbuntu kernel: [15256.929009] douane:1162:exit_module: Kernel module removed!
May  3 14:12:34 zUbuntu kernel: [15256.929130] [leak_check] Target module: "douane", init area at ffffffffa0005000, core area at ffffffffa04e4000
May  3 14:12:34 zUbuntu kernel: [15256.929144] [leak_check] Totals:
May  3 14:12:34 zUbuntu kernel: [15256.929148] [leak_check] Allocations: 1786674
May  3 14:12:34 zUbuntu kernel: [15256.929151] [leak_check] Possible leaks: 0
May  3 14:12:34 zUbuntu kernel: [15256.929154] [leak_check] Unallocated frees: 0
May  3 14:12:34 zUbuntu kernel: [15256.929156] [leak_check] ======== end of LeakCheck report ========

With the daemon enable (and no freeze):

May  3 14:25:56 zUbuntu kernel: [16059.298246] douane:1162:exit_module: Kernel module removed!
May  3 14:25:56 zUbuntu kernel: [16059.298384] [leak_check] Target module: "douane", init area at ffffffffa0005000, core area at ffffffffa04ea000
May  3 14:25:56 zUbuntu kernel: [16059.298398] [leak_check] Totals:
May  3 14:25:56 zUbuntu kernel: [16059.298402] [leak_check] Allocations: 1789431
May  3 14:25:56 zUbuntu kernel: [16059.298406] [leak_check] Possible leaks: 0
May  3 14:25:56 zUbuntu kernel: [16059.298409] [leak_check] Unallocated frees: 0
May  3 14:25:56 zUbuntu kernel: [16059.298411] [leak_check] ======== end of LeakCheck report ========

So it looks like there's no memory leaks in the kernel module for now.
In the other hand I saw that the daemon process was using 50% of a core of my CPU during the copy, and the memory increased a lot, so we have to fix this also.

@zedtux
Owner

@pavlinux in the case you would like to use KEDR framework while working on the LKM here is a blog article from my blog which explain how I make it working: http://blog.zedroot.org/debug-your-linux-kernel-module/.

@pavlinux

For exapmle, leaks before this commit 881dbec difficult catching.
Without intentional attack probability occurrence empty TCP or UDP header, with the already established IP connection, about a little less than zero. :)

@pavlinux

About leaks in push() function:
1. nlmsg_put() - this function creates and fills the header future NETLINK packet. That returned NULL should really try.
2. After netlink_unicast() do not need to release the memory!
3. After https://github.com/Douane/douane-dkms/blob/master/douane.c#L443 сorrect call nlmsg_cancel(skb, nlh), then kfree_skb(skb)

@zedtux
Owner

Thank you @pavlinux for the details.

I have installed now everything in order to debug the kernel module. I'm going to work on it in order to make the douane crashing and find the issue.
(I invite you to do the same when patching the kernel module).

@hotice

I had another panic but the kernel log is pretty large (about 120mb). If you still need this, I can post it here...

As for the packaging, what package has the wrong license? I don't use Launchpad recipes to build the packages, I upload them myself... And about the daemon, indeed it should be stopped when it's uninstalled, but it shouldn't be started when it's installed since it's started by the Configurator... I'll fix the packaging after this bug is fixed...

@pavlinux

@hotice How does this relate douane? In previous logs you falls avahi and apparmor, nmdb and apparmor, ... And turn off the flag -DEBUG in module. There's nothing useful.

@hotice

Well, if I stop the Douane daemon and uninstall douane-dkms, I don't get any kernel panics so... Besides, zedtux was able to reproduce it too.

@pavlinux

@hotice You can work, one day, without apparmor? Apparmor uses a subsystem "audit", which also uses the Netlink. I think they are in conflict.

@hotice

Ubuntu uses Apparmor by default but I'll disable it and see if this still occurs. Completely removing apparmor is not possible because many Ubuntu packages depend on it...

@hotice

No kernel panics so far but my laptop froze twice. The last time, I was using Firefox and downloading 2 torrents using Transmission.

Because this is also my work laptop and I don't have another computer, I can't continue testing this... I simply can't work with my laptop freezing and gettting kernel panics all day long so I had to completely remove douane for now.

@zedtux
Owner

Ok, on my side, I'm going to check the kernel panic/freezes thanks to kdump... I'll let you know when I have some news.

@hotice sorry for this ...

@hotice

@zedtux I don't have to take @pavlinux 's crap so I will stop posting comments here or report any other bug. If you fix this, you have my email...

@hotice

By the way, the file was so large because Douane with debugging enabled spammed the hell out of the log files.

@zedtux
Owner

@hotice ok ok, no problem. For me the same, with the debug enabled, I had a kern.log file with a size of 7.5Go !

As soon as I have some news, I'll come back to you guys.

@zedtux
Owner

@pavlinux if you're interested, here is my blog article about kdump: http://blog.zedroot.org/linux-kernel-debuging-using-kdump-and-crash/.

@pavlinux

@zedtux Why kdump, why crash, why KEDR, you have the source code. :) There you can see everything.

@zedtux
Owner

@pavlinux well I see what you mean :) But when my computer freeze, if I can't see the issue, it's then harder to fix it, isn't it ?

@zedtux
Owner

Short message to inform that I'm waiting my compute to crash (with Kdump enabled) ... but since 2 days it is still working ...

@zedtux
Owner

As of today, and as far as I know, 3 people are impacted by the kernel freeze issue.

I'm fine to have it reported but it doesn't help to have multiple issues opened in the different repos (and I understand the issue to know where to open the issue, even if I have written a wiki page about to know where to open an issue).

This said, I'm going to close any other tickets opened regarding kernel freeze and redirect people to this issue.

@ho-mo

My system crashed now agian ! what your distro ?!
I am use Ubuntu 14.04 x64
execuse me I dont know , That this Problem related to douane-dkms !
This in my kenl.log
Sep 21 01:36:50 double kernel: [ 1528.529311] r8169 0000:02:00.0 eth0: link down
Sep 21 01:36:50 double kernel: [ 1528.529350] r8169 0000:02:00.0 eth0: link down
Sep 21 01:36:50 double kernel: [ 1528.529363] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 21 01:36:52 double kernel: [ 1530.725048] r8169 0000:02:00.0 eth0: link up
Sep 21 01:36:52 double kernel: [ 1530.725064] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 21 01:42:29 double kernel: [ 1867.586261] douane:414:push: BLOCKED PUSH: process_path is blank.
Sep 21 01:42:29 double kernel: [ 1867.602276] douane:414:push: BLOCKED PUSH: process_path is blank.
Sep 21 01:42:29 double kernel: [ 1867.615575] douane:414:push: BLOCKED PUSH: process_path is blank.
Sep 21 01:42:29 double kernel: [ 1867.628695] douane:414:push: BLOCKED PUSH: process_path is blank.
Sep 21 01:42:29 double kernel: [ 1867.876567] douane:414:push: BLOCKED PUSH: process_path is blank.
Sep 21 01:42:29 double kernel: [ 1867.889773] douane:414:push: BLOCKED PUSH: process_path is blank.
Sep 21 01:42:29 double kernel: [ 1867.903135] douane:414:push: BLOCKED PUSH: process_path is blank.
Sep 21 01:42:29 double kernel: [ 1867.916129] douane:414:push: BLOCKED PUSH: process_path is blank.
Sep 21 01:42:29 double kernel: [ 1867.917577] douane:414:push: BLOCKED PUSH: process_path is blank.
Sep 21 01:42:36 double kernel: [ 1874.247981] douane:457:push: Message ignored as Netfiler socket is busy.
Sep 21 01:42:36 double kernel: [ 1874.248071] ------------[ cut here ]------------
Sep 21 01:42:36 double kernel: [ 1874.248112] kernel BUG at /build/buildd/linux-3.13.0/mm/slub.c:3365!
Sep 21 01:42:36 double kernel: [ 1874.248158] invalid opcode: 0000 [#1] SMP
Sep 21 01:42:36 double kernel: [ 1874.248190] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 xt_CHECKSUM iptable_mangle bridge stp llc ebtable_nat ebtables douane(OF) pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) bnep rfcomm bluetooth binfmt_misc ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm gpio_ich snd_page_alloc snd_seq_midi coretemp snd_seq_midi_event kvm_intel kvm snd_rawmidi serio_raw snd_seq lpc_ich snd_seq_device snd_timer snd ppdev parport_pc lp soundcore parport mac_hid btrfs xor raid6_pq libcrc32c hid_generic usbhid hid r8169 i915 mii floppy video i2c_algo_bit drm_kms_helper drm

doual core 3Ghz Intel
2 GB RAM
Intel Onboard Graphic
gnome-shell interface
when i start douane with this command
sudo service douane start
and Use foxyproxy on firefox 127.0.0.1:9050 (socks Tor proxy) and i went to this sites
jadi.net , https://epayment.bmi.ir/Remain
system crashed

@zedtux
Owner

execuse me I dont know , That this Problem related to douane-dkms !

No worries ;-).

I'm using also Ubuntu 14.04 64 bits, but since I have installed kdump and crash in order to collect crash information, my system haven't crashed.
If you could try to follow the instruction on my blog : http://blog.zedroot.org/linux-kernel-debuging-using-kdump-and-crash/.
In case you succeed, send me the crash file and I can analyse it.

@arlsr

Doune is also unusable for me due to system freezes/crashes.

@zedtux
Owner

Thank you @arlsr to join us. As of now, multiple people has issues, but no one of you are able to price me wugh a crash file.
Without it, I can't do anything.

May I ask someone of you to follow the instructions I have already provided and send me the crash file!?

Thank you in advance.

@zedtux
Owner

Any chance @arlsr that you provide me with a crash file as described on my blog ?

@arlsr

A few reboots have damaged my system in the past, I'm hesitant to try again. I may in the future when I have more time though. I'm considering stripping Douane down to the fundamentals to see if I can find the cause of the crashes.

@zedtux
Owner

My blog article is describing how to use kdump which will boot a second kernel when the main will crash so that your computer is going to move to the new running kernel and don't loose data.

Anyway what about using a Virtual Machine ?

@zedtux zedtux referenced this issue in Douane/Douane
Closed

Problem with goagent #37

@arlsr

I've set up kdump. My computer doesn't reboot when it crashes though, it hangs and no dump is created. Even when triggering the test crash.

@themighty1

Same here. I didnt bother to report though because I thought that was the case for everyone. I ran Douanne on a fresh 14.04 install in Virtualbox. It would take only a few minutes before the machine locked up.

@zedtux
Owner

That's wired. I gave a try to my article after having published it in order to be sure all is working well. Your computer should not restart and you should have a file in /var/crash/ (something like this).

@arlsr

There's only kexec_cmd in /var/crash.

cat /proc/cmdline shows crashkernel=384M-:128M

$ cat /sys/kernel/kexec_crash_loaded 
1

I have a large amount of memory and no swap, that might be the issue.

@zedtux
Owner

I've switched to Debian and I'm considering trying Douane in order to finally fix this issue.

@junkvolny

Yesterday, all I had to do for system freeze was run web browser. Today it worked somehow without freeze for quite some time, but I was able to force system hang with following:
1. open web browser (i used firefox), that has allow rule in place, web page loads
2. stop the daemon with #sudo /etc/init.d/douane stop
3. If not already system hang, repeat deamon start/stop few times, mix in few refreshes in web browser.
Works like charm (irony: system freeze works).

Maybe I should try to watch log of kernel module, not just daemon with #tail -f /var/log/douane.log but not sure how and if it's enabled by default, in any case, if you can reproduce freeze with these steps, you might be better at it.

@junkvolny

I did huge amount of searching today for possible alternatives today, and I found GPLed project http://sourceforge.net/projects/leopardflower/ that uses iptables and libnetfilter_queue, thus it does not require kernel module (!). I must admit I have not tested it much. I think all these troubles are because kernel module is so hard to debug.

Your "Douane personal firewall" and mentioned "Leopard flower per-application (personal) firewall" are the only application firewalls for current Linux/Ubuntu out there, I am afraid.

@arlsr

leopardflower has plenty of issues of its own sadly, but none that crash the host.

@zedtux
Owner

As I'm now using Debian (no more Ubuntu), I will now test it with Debian.

@zedtux
Owner

Alright, I have a working Debian in VirtualBox (I have updated the dependencies wiki page for Debian) and I got the crash and the crash dump file has been generated.

I'm giving a look now.

@zedtux
Owner

OK now I see the issue. The freeze come from an issue in the kernel module communication with the daemon.

Simple things like ping are working like a charm while a lot of activity is making the freeze happening. I will try to solve this in my free time...

@zedtux
Owner

@junkvolny @arlsr @ho-mo I have created the branch fix-kernel-freeze where I did modification which allowed me to use Douane in a VM. After 15m the VM slow down and the douane-dialog process was using 100% of the CPU so I need to have a look to this but at least the kernel freeze is gone for now.

@junkvolny

I tested today, on same virtual machine that was freezing before, and yes, it seems the kernel freeze is gone, well done.

I didn't test long run, for dialog-process freeze, but to be honest, dialog is least important component - it cannot crash kernel + if it crashes, it does not stop daemon from enforcing rules, hopefully.

@zedtux
Owner

... if it crashes, it does not stop daemon from enforcing rules, hopefully.

You're right, it do not.

I have already identified in the issue Douane/douane-daemon#6 that the daemon is increasing the memory usage and after an amount of time is using all the memory so I need to fix it.

Anyway thank you @junkvolny for your confirmation. I would like to have another confirmation before to close this issue. Can one of you @arlsr, @ho-mo or @hotice confirm that the kernel freeze is gone and I can close this issue ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.