dsc stopped writing reports with 2.3.0

[McStork]

Hello and happy new year !

We installed dsc 2.3.0 on two servers shortly after the version release. Three days later, the daemon of one of the servers stopped writing the reports (but dsc kept running). The same situation occurred on the second server, 10 days after the installation of dsc 2.3.0.
It seems that no logs were written when the situations occurred.

We performed today a rollback to 2.1.1 which was the previously installed version on these servers.

[jelu]

Thanks for the report, there seem to be issues with the threading, signaling and forking. Working on it but hard to say when it can be ready for testing.

What distribution are you running?

You can try running with the -T flag that will disable the usage of threads.

[McStork]

Thanks for the quick reply :-).
We are running Debian Jessie x64 on both servers.

[jelu]

Can you give me the spec of the servers and the QPS your receiving?

[McStork]

Servers receive around 2 to 3 QPS.
They are LXC containers with 4 cpu @ 2.80GHz and 4GB RAM each.

[jelu]

Can you describe how it stopped? What did you see on the ps output? Did you do a strace?

[McStork]

ps showed that the process was still running even though reports stopped being written. We did not run strace :-(.

[jelu]

What was the last file in the data directory? An ...xml.XXX?

[McStork]

No, the last written file had the proper .xml / .json extension.

[jelu]

If you have the possibility could you setup another container and run the develop branch and generate some QPS to it?

It seems that the issue is related to threads and that the process forks when writing the files, this makes libraries very rarely lock up. I was lucky enough to catch this while stracing and I saw a lockup in the NSS library used to get the IP protocol name.

The latest commits to develop has been to ensure signal handlers are correct after fork and to use thread safe libc functions.

I am also setting up dsc on all our build VMs to run the latest develop branch build continuously, generating 10 QPS against them and will monitor file creation.

[jelu]

If you have not already please read my announcement regarding this:
https://lists.dns-oarc.net/pipermail/dsc/2017-January/000361.html

I have setup some test now for dsc, if you have any other ideas of tests please let me know:
https://dev.dns-oarc.net/jenkins/view/dsctest/

[jelu]

Are you able to test the latest develop branch?

[McStork]

Hi @jelu. We haven't tested the develop branch yet. But we might be able to do so next week.

[McStork]

We now have it running on a server.

[jelu]

Which commit are you running?

[McStork]

fa191b0

[jelu]

Do you have any issue to report with the develop branch?

[McStork]

We had no issue with the develop branch so far.

[McStork]

Thanks @jelu! Good job. :-)