New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement on ommongodb: mongo-c-driver with TLS and replicaset #1917

Merged
merged 39 commits into from Nov 23, 2017

Conversation

Projects
None yet
5 participants
@jjourdin
Contributor

jjourdin commented Oct 27, 2017

Hello,

We propose improvements on the existing ommongodb output module. The major improvement is the use of the mongo-c-driver library instead of the deprecated mongo-client library.

New features are :

  • Handle TLS connexion to mongodb
  • Handle MongoDB Replicasets
  • Added the 'ssl_ca' and 'ssl_cert' directives to configure tls connexion
  • Added 'uristr' directive to configure the connexion uri in the form of 'mongodb://...'

Example configuration:

action(
     type="ommongodb" 
     uristr="mongodb://vulture:9091,vulture2:9091/?replicaset=Vulture&ssl=true" 
     ssl_cert="/var/db/mongodb/mongod.pem" 
     ssl_ca="/var/db/mongodb/ca.pem" 
     db="logs" 
     collection="syslog")

The "uristr" directive overrides the 'uid', 'pwd', 'server' and 'serverport' directives. To create the uri, please refer to the official MongoDB Connection String documentation.

uid, pwd, server and serverport directives are useless now, but still supported in this module, to preserve compatibility with existing configuration.

Feel free to contact us at aDvens if you have any questions: jeremie.jourdin@advens.fr / hugo.soszynski@advens.fr

Regards,

Hugo & Jérémie

Hugo SOSZYNSKI added some commits Oct 26, 2017

Hugo SOSZYNSKI
Updated ommongodb to use mongo-c-driver which offers a lot more funct…
…ions such as ssl connexions and MongoDB Replicasets compatibility.

 - Added the 'ssl_ca' and 'ssl_cert' directives to configure ssl connexion.
 - Added 'uristr' directive to configure the connexion uri directly. This directive overrides the 'uid', 'pwd', 'server' and 'serverport' directives. To create the uri, please refer to the official MongoDB Connection String documentation.
Hugo SOSZYNSKI
@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards
Member

rgerhards commented Oct 27, 2017

@jjourdin jjourdin changed the title from Improvement on ommongodb: mongo-c-driver with, TLS, replicaset to Improvement on ommongodb: mongo-c-driver with TLS and replicaset Oct 27, 2017

@rgerhards rgerhards added this to the v8.31 milestone Oct 27, 2017

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Oct 27, 2017

Member

Feel free to contact us at aDvens if you have any question

Do you also watch the github issue or is email required?

Member

rgerhards commented Oct 27, 2017

Feel free to contact us at aDvens if you have any question

Do you also watch the github issue or is email required?

@jjourdin

This comment has been minimized.

Show comment
Hide comment
@jjourdin

jjourdin Oct 27, 2017

Contributor

We will update the documentation. If there are any issue feel free to assign to me.

Contributor

jjourdin commented Oct 27, 2017

We will update the documentation. If there are any issue feel free to assign to me.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Oct 28, 2017

Member

a rebase to current master will make that current freebsd CI failure go away

Member

rgerhards commented Oct 28, 2017

a rebase to current master will make that current freebsd CI failure go away

@HugoSoszynski

This comment has been minimized.

Show comment
Hide comment
@HugoSoszynski

HugoSoszynski Nov 2, 2017

Contributor

Hello,

After reading the logs of the failed builds on Travis it seems that they failed due to the installation of libmongo-client-dev instead of libmongoc-dev in the build environment: libmongoc being the new dependency needed for ommongodb as specified in the merge request message.

Do we need to make any change or is it up to you to do so ?

Regards,

Hugo.

Contributor

HugoSoszynski commented Nov 2, 2017

Hello,

After reading the logs of the failed builds on Travis it seems that they failed due to the installation of libmongo-client-dev instead of libmongoc-dev in the build environment: libmongoc being the new dependency needed for ommongodb as specified in the merge request message.

Do we need to make any change or is it up to you to do so ?

Regards,

Hugo.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 2, 2017

Member

Do we need to make any change or is it up to you to do so ?

You have control over travis. It's probably installed in .travis.yml in the root or here: https://github.com/rsyslog/rsyslog/blob/master/tests/travis/install.sh

Member

rgerhards commented Nov 2, 2017

Do we need to make any change or is it up to you to do so ?

You have control over travis. It's probably installed in .travis.yml in the root or here: https://github.com/rsyslog/rsyslog/blob/master/tests/travis/install.sh

@HugoSoszynski

This comment has been minimized.

Show comment
Hide comment
@HugoSoszynski

HugoSoszynski Nov 3, 2017

Contributor

Hello,

As long as the tests are done on Ubuntu 14.04 LTS and the libmongoc-dev package is not available on this version, is it okay to fetch the mongo-c-driver Github repository and compile it for the build ?

I know this solution is not perfect but after some research, it looks like this is a rare case and most distributions now includes the package.

Regards,

Hugo.

Contributor

HugoSoszynski commented Nov 3, 2017

Hello,

As long as the tests are done on Ubuntu 14.04 LTS and the libmongoc-dev package is not available on this version, is it okay to fetch the mongo-c-driver Github repository and compile it for the build ?

I know this solution is not perfect but after some research, it looks like this is a rare case and most distributions now includes the package.

Regards,

Hugo.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 3, 2017

Member

Yeah, it's unfortunately increasingly becoming a problem that the Travis images are too outdated. I'd say let's build the dependency if we do not have a package. We just need to check that we do not do that in too many parallel builds, because of runtime limits in Travis.

Member

rgerhards commented Nov 3, 2017

Yeah, it's unfortunately increasingly becoming a problem that the Travis images are too outdated. I'd say let's build the dependency if we do not have a package. We just need to check that we do not do that in too many parallel builds, because of runtime limits in Travis.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 5, 2017

Member

@HugoSoszynski I am also working on getting the ubuntu16 buildbot slave in shape to compile ommongodb. It provides the best clang static analyzer output.

Member

rgerhards commented Nov 5, 2017

@HugoSoszynski I am also working on getting the ubuntu16 buildbot slave in shape to compile ommongodb. It provides the best clang static analyzer output.

@HugoSoszynski

This comment has been minimized.

Show comment
Hide comment
@HugoSoszynski

HugoSoszynski Nov 9, 2017

Contributor

Hello,

We are currently testing with our Travis. While testing, we have a problem with the libbson embedded with the mongo-c-driver. Libbson is not able to pass its ./configure because it's unable to find the pthread library necessary on non-windows environments.

  • The libpthread-stubs0-dev is installed.
  • The ./configure works without any problem on an Ubuntu 14 VM I use for local testing.
  • The configure.ac and the .m4 associated strictly follow the autotools documentation.
  • I did not found any working solution on Google.

This is becoming kind of a problem.
As rsyslog and some of its plugins uses pthread, I wondered if you had any idea.

Regards.

Contributor

HugoSoszynski commented Nov 9, 2017

Hello,

We are currently testing with our Travis. While testing, we have a problem with the libbson embedded with the mongo-c-driver. Libbson is not able to pass its ./configure because it's unable to find the pthread library necessary on non-windows environments.

  • The libpthread-stubs0-dev is installed.
  • The ./configure works without any problem on an Ubuntu 14 VM I use for local testing.
  • The configure.ac and the .m4 associated strictly follow the autotools documentation.
  • I did not found any working solution on Google.

This is becoming kind of a problem.
As rsyslog and some of its plugins uses pthread, I wondered if you had any idea.

Regards.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 9, 2017

Member

As rsyslog and some of its plugins uses pthread, I wondered if you had any idea.

This is how rsyslog checks for pthreads: https://github.com/rsyslog/rsyslog/blob/master/configure.ac#L344

We never had a problem finding the lib. Maybe the libbson configure.ac needs a fix?

Member

rgerhards commented Nov 9, 2017

As rsyslog and some of its plugins uses pthread, I wondered if you had any idea.

This is how rsyslog checks for pthreads: https://github.com/rsyslog/rsyslog/blob/master/configure.ac#L344

We never had a problem finding the lib. Maybe the libbson configure.ac needs a fix?

Hugo SOSZYNSKI added some commits Nov 10, 2017

@HugoSoszynski

This comment has been minimized.

Show comment
Hide comment
@HugoSoszynski

HugoSoszynski Nov 16, 2017

Contributor

Hello,

I would like to know if there is a way to access the buildbot logs please.
Clicking on the Details link is not working on my side (just an infinite loading in my browser) and after a check at the contributing doc, I saw nothing about this.

Thank you.

Contributor

HugoSoszynski commented Nov 16, 2017

Hello,

I would like to know if there is a way to access the buildbot logs please.
Clicking on the Details link is not working on my side (just an infinite loading in my browser) and after a check at the contributing doc, I saw nothing about this.

Thank you.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 16, 2017

Member

I would like to know if there is a way to access the buildbot logs please.

it's running on port 8010 - do you block this?

Member

rgerhards commented Nov 16, 2017

I would like to know if there is a way to access the buildbot logs please.

it's running on port 8010 - do you block this?

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 16, 2017

Member

I also had a quick look at the CentOS 7 buildbot slave. Libmongoc-dev is missing there too. Trying yum install libmongoc-dev libmongoc-devel does not bring up a package, nor does yum search. Also missing there? Or can you suggest an install source. I guess same story for Centos 6...

Member

rgerhards commented Nov 16, 2017

I also had a quick look at the CentOS 7 buildbot slave. Libmongoc-dev is missing there too. Trying yum install libmongoc-dev libmongoc-devel does not bring up a package, nor does yum search. Also missing there? Or can you suggest an install source. I guess same story for Centos 6...

@HugoSoszynski

This comment has been minimized.

Show comment
Hide comment
@HugoSoszynski

HugoSoszynski Nov 16, 2017

Contributor

Thank you, I will soon have access to the buildbot.
The libmongoc-dev is not available in the default repositories, it requires the use of repositories from the Fedora project.
I'm waiting to have a look to the builbot to find a generic solution.

Contributor

HugoSoszynski commented Nov 16, 2017

Thank you, I will soon have access to the buildbot.
The libmongoc-dev is not available in the default repositories, it requires the use of repositories from the Fedora project.
I'm waiting to have a look to the builbot to find a generic solution.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 16, 2017

Member

The libmongoc-dev is not available in the default repositories, it requires the use of repositories from the Fedora project.

I enabled what is safe to enable on CentOS 7: yum --enablerepo=extras install epel-release Unfortunately, they do not contain the package. Will look to build manually.

Member

rgerhards commented Nov 16, 2017

The libmongoc-dev is not available in the default repositories, it requires the use of repositories from the Fedora project.

I enabled what is safe to enable on CentOS 7: yum --enablerepo=extras install epel-release Unfortunately, they do not contain the package. Will look to build manually.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 16, 2017

Member

ok, from source bails out due to missing g++, which I can't find for CentOS 7. So as it looks, we unfortunately cannot even build-test it on CentOS 7. That's a bit of a shame as that platform is frequently used. If you have an idea how to build it without diverging too much from a "save" CentOS base, I am all ears.

Member

rgerhards commented Nov 16, 2017

ok, from source bails out due to missing g++, which I can't find for CentOS 7. So as it looks, we unfortunately cannot even build-test it on CentOS 7. That's a bit of a shame as that platform is frequently used. If you have an idea how to build it without diverging too much from a "save" CentOS base, I am all ears.

@deoren

This comment has been minimized.

Show comment
Hide comment
@deoren

deoren Nov 16, 2017

Contributor

@rgerhards: ok, from source bails out due to missing g++, which I can't find for CentOS 7.

Looks like you need the gcc-c++ package:

# yum provides g++

Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.wdc1.us.leaseweb.net
 * extras: reflector.westga.edu
 * updates: mirror.jax.hugeserver.com
base/7/x86_64/filelists_db                                                                                                                                                                  | 6.7 MB  00:00:04     
extras/7/x86_64/filelists_db                                                                                                                                                                | 494 kB  00:00:00     
updates/7/x86_64/filelists_db                                                                                                                                                               | 2.1 MB  00:00:00     
gcc-c++-4.8.5-16.el7.x86_64 : C++ support for GCC
Repo        : base
Matched from:
Filename    : /usr/bin/g++
Contributor

deoren commented Nov 16, 2017

@rgerhards: ok, from source bails out due to missing g++, which I can't find for CentOS 7.

Looks like you need the gcc-c++ package:

# yum provides g++

Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.wdc1.us.leaseweb.net
 * extras: reflector.westga.edu
 * updates: mirror.jax.hugeserver.com
base/7/x86_64/filelists_db                                                                                                                                                                  | 6.7 MB  00:00:04     
extras/7/x86_64/filelists_db                                                                                                                                                                | 494 kB  00:00:00     
updates/7/x86_64/filelists_db                                                                                                                                                               | 2.1 MB  00:00:00     
gcc-c++-4.8.5-16.el7.x86_64 : C++ support for GCC
Repo        : base
Matched from:
Filename    : /usr/bin/g++
@HugoSoszynski

This comment has been minimized.

Show comment
Hide comment
@HugoSoszynski

HugoSoszynski Nov 16, 2017

Contributor

So I looked at the logs from the ubuntu16 build.
ommongodb cannot compile because Ubuntu 16 uses an outdated version of libmongoc: 1.3.1 and we are now at version 1.8.1.
Those versions have a difference in the fields of the mongoc_ssl_opt_t structure. Not compiling because of a scalar initialization of this particular struct.

I will fix the code to work with both versions as long as we do not use this specific field.

Contributor

HugoSoszynski commented Nov 16, 2017

So I looked at the logs from the ubuntu16 build.
ommongodb cannot compile because Ubuntu 16 uses an outdated version of libmongoc: 1.3.1 and we are now at version 1.8.1.
Those versions have a difference in the fields of the mongoc_ssl_opt_t structure. Not compiling because of a scalar initialization of this particular struct.

I will fix the code to work with both versions as long as we do not use this specific field.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 16, 2017

Member

Looks like you need the gcc-c++ package:

Doh! I thought I had tried that, but... Now it's there. I'll modify the buildbot conf so that Centos7 will also try to build mongodb. Just so that you know should a new error pop up there (oh, the joys of multi-distro development... ;-))

Member

rgerhards commented Nov 16, 2017

Looks like you need the gcc-c++ package:

Doh! I thought I had tried that, but... Now it's there. I'll modify the buildbot conf so that Centos7 will also try to build mongodb. Just so that you know should a new error pop up there (oh, the joys of multi-distro development... ;-))

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 16, 2017

Member

re-initiated Centos7 build. If it fails, I'll check if I screwed up with my CI updates and will fix that. I'll let you know in any case.

Member

rgerhards commented Nov 16, 2017

re-initiated Centos7 build. If it fails, I'll check if I screwed up with my CI updates and will fix that. I'll let you know in any case.

@HugoSoszynski

This comment has been minimized.

Show comment
Hide comment
@HugoSoszynski

HugoSoszynski Nov 17, 2017

Contributor

It looks like the missing lib on Centos7 is libmongo-client, this is not the lib used by ommongodb but it is used for the logctl usertool.

Contributor

HugoSoszynski commented Nov 17, 2017

It looks like the missing lib on Centos7 is libmongo-client, this is not the lib used by ommongodb but it is used for the logctl usertool.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 17, 2017

Member

@HugoSoszynski I think we should not use multiple mongo libraries, that will probably result in a mess. So logctl needs to be updated as well.

Member

rgerhards commented Nov 17, 2017

@HugoSoszynski I think we should not use multiple mongo libraries, that will probably result in a mess. So logctl needs to be updated as well.

Hugo SOSZYNSKI added some commits Nov 17, 2017

Hugo SOSZYNSKI
Corrected some compilation warning and errors.
No longer installing libmongo-client in Travis script.
Hugo SOSZYNSKI
Added compilation time mongo-c-driver version check to not use deprec…
…ated mongoc function when newer are available in the logctl usertool.
Hugo SOSZYNSKI
Added __attribute__((unused)) in logctl usertool to silence warning t…
…hat occurs with use of mongo-c-driver >= 1.5.0. This is needed because stable versions of Debian based distributions uses mongo-c-driver <= 1.4.2.
Hugo SOSZYNSKI
Made the funtions static hopping to correct the "no previous prototyp…
…e" problème and suppresion of trailing spaces
@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 23, 2017

Member

Congrats! Special thanks to @HugoSoszynski for the presistence in driving this forward - much appreciated. I am sure this is a great addition to rsyslog.

Member

rgerhards commented Nov 23, 2017

Congrats! Special thanks to @HugoSoszynski for the presistence in driving this forward - much appreciated. I am sure this is a great addition to rsyslog.

@rgerhards rgerhards merged commit 0fe2de8 into rsyslog:master Nov 23, 2017

14 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
rsyslog centos6 rsyslog Build done
Details
rsyslog centos7 rsyslog Build done
Details
rsyslog debian rsyslog Build done
Details
rsyslog fedora23x32 rsyslog Build done
Details
rsyslog fedora26x64 rsyslog Build done
Details
rsyslog freebsd rsyslog Build done
Details
rsyslog solaris10sparc rsyslog Build done
Details
rsyslog solaris10x64 rsyslog Build done
Details
rsyslog solaris11sparc rsyslog Build done
Details
rsyslog solaris11x64 rsyslog Build done
Details
rsyslog suse rsyslog Build done
Details
rsyslog ubuntu rsyslog Build done
Details
rsyslog ubuntu16 rsyslog Build done
Details
@algernon

This comment has been minimized.

Show comment
Hide comment
@algernon

algernon Nov 23, 2017

As rsyslog was the last big user of libmongo-client, thank you @HugoSoszynski for this work! Will make it easier to properly retire the library and eventually drop it from distributions. \o/

algernon commented Nov 23, 2017

As rsyslog was the last big user of libmongo-client, thank you @HugoSoszynski for this work! Will make it easier to properly retire the library and eventually drop it from distributions. \o/

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Nov 23, 2017

Member

@algernon ... and thanks for developing libmongo-client in the first place! Was much appreciated.

Member

rgerhards commented Nov 23, 2017

@algernon ... and thanks for developing libmongo-client in the first place! Was much appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment