New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mmanon plugin doesn't support IPv6 #1614

Closed
mejo- opened this Issue Jun 13, 2017 · 29 comments

Comments

Projects
None yet
7 participants
@mejo-

mejo- commented Jun 13, 2017

Hello,

I just discovered that the mmamon IP address anonymization plugin for rsyslog doesn't support IPv6. I found a blog post by Rainer Gerhards from 2013 where he mentions that "IPv6 support is planned".

@rgerhards: Do you still work on that? Do you have a preliminary version available for testing?

Im asking as IPv6 is (slowly) becoming reality ;)

@xshadow

This comment has been minimized.

Show comment
Hide comment
@xshadow

xshadow Jun 14, 2017

I'm also looking for a solution with mmanon with IPv6, would be happy to test.

xshadow commented Jun 14, 2017

I'm also looking for a solution with mmanon with IPv6, would be happy to test.

@xshadow

This comment has been minimized.

Show comment
Hide comment
@xshadow

xshadow Jul 15, 2017

@rgerhards Could you provide the current state of the code? Else I would think about to file a bug bounty.

xshadow commented Jul 15, 2017

@rgerhards Could you provide the current state of the code? Else I would think about to file a bug bounty.

@davidelang

This comment has been minimized.

Show comment
Hide comment
@davidelang

davidelang Jul 15, 2017

Contributor
Contributor

davidelang commented Jul 15, 2017

@mejo-

This comment has been minimized.

Show comment
Hide comment
@jgerhards

This comment has been minimized.

Show comment
Hide comment
@jgerhards

jgerhards Aug 11, 2017

Contributor

How would you like to anonymize the address? are there any parameters you think are neccessary, i.e. how may bits are to be anonymized?

Contributor

jgerhards commented Aug 11, 2017

How would you like to anonymize the address? are there any parameters you think are neccessary, i.e. how may bits are to be anonymized?

@mejo-

This comment has been minimized.

Show comment
Hide comment
@mejo-

mejo- Aug 11, 2017

I would expect all bits to be anonymized, e.g. by replacing it with 0:0:0:0:0:0:0:0. Just like the mmanon plugin does for IPv4 right now.

mejo- commented Aug 11, 2017

I would expect all bits to be anonymized, e.g. by replacing it with 0:0:0:0:0:0:0:0. Just like the mmanon plugin does for IPv4 right now.

@jgerhards

This comment has been minimized.

Show comment
Hide comment
@jgerhards

jgerhards Aug 11, 2017

Contributor

In that case i'll try to implement an ipv6 anonymizer similar to the ipv4 one. After i'm done with that, I might put in some more work.

Contributor

jgerhards commented Aug 11, 2017

In that case i'll try to implement an ipv6 anonymizer similar to the ipv4 one. After i'm done with that, I might put in some more work.

@mejo-

This comment has been minimized.

Show comment
Hide comment
@mejo-

mejo- Aug 11, 2017

@jgerhards: that would be awesome. You will do privacy-aware admins and their users a big favour. Log anonymization is an important step for data minimization.

mejo- commented Aug 11, 2017

@jgerhards: that would be awesome. You will do privacy-aware admins and their users a big favour. Log anonymization is an important step for data minimization.

@davidelang

This comment has been minimized.

Show comment
Hide comment
@davidelang

davidelang Aug 12, 2017

Contributor
Contributor

davidelang commented Aug 12, 2017

@jgerhards

This comment has been minimized.

Show comment
Hide comment
@jgerhards

jgerhards Aug 15, 2017

Contributor

quick update: I have found some bugs in the ipv4 anonymizer that I'll fix before getting back to work on the ipv6 one.

Contributor

jgerhards commented Aug 15, 2017

quick update: I have found some bugs in the ipv4 anonymizer that I'll fix before getting back to work on the ipv6 one.

@mejo-

This comment has been minimized.

Show comment
Hide comment
@mejo-

mejo- Aug 16, 2017

Thanks at @jgerhards for working on it.

mejo- commented Aug 16, 2017

Thanks at @jgerhards for working on it.

@jgerhards

This comment has been minimized.

Show comment
Hide comment
@jgerhards

jgerhards Aug 16, 2017

Contributor

I have now started a meta issue tracker for the ipv4 bugs (and rewrite) as well as ipv6 function.

However, please don't close this issue since it will still be in use for the initial ipv6 anonymization feature request.

Contributor

jgerhards commented Aug 16, 2017

I have now started a meta issue tracker for the ipv4 bugs (and rewrite) as well as ipv6 function.

However, please don't close this issue since it will still be in use for the initial ipv6 anonymization feature request.

@jgerhards

This comment has been minimized.

Show comment
Hide comment
@jgerhards
Contributor

jgerhards commented Aug 24, 2017

@xshadow

This comment has been minimized.

Show comment
Hide comment
@xshadow

xshadow Aug 24, 2017

Sounds pretty awesome! Thanks

xshadow commented Aug 24, 2017

Sounds pretty awesome! Thanks

jgerhards added a commit to jgerhards/rsyslog that referenced this issue Sep 1, 2017

jgerhards added a commit to jgerhards/rsyslog that referenced this issue Sep 1, 2017

jgerhards added a commit to jgerhards/rsyslog that referenced this issue Sep 2, 2017

jgerhards added a commit to jgerhards/rsyslog that referenced this issue Sep 2, 2017

jgerhards added a commit to jgerhards/rsyslog that referenced this issue Sep 2, 2017

jgerhards added a commit to jgerhards/rsyslog that referenced this issue Sep 2, 2017

jgerhards added a commit to jgerhards/rsyslog that referenced this issue Sep 2, 2017

@jgerhards

This comment has been minimized.

Show comment
Hide comment
@jgerhards

jgerhards Sep 4, 2017

Contributor

I have now completed implementing IPv6 support for mmanon. It now just needs to be merged. You can find out more here: http://jan.gerhards.net/2017/09/mmanon-rewrite-finished-for-time-being.html

Contributor

jgerhards commented Sep 4, 2017

I have now completed implementing IPv6 support for mmanon. It now just needs to be merged. You can find out more here: http://jan.gerhards.net/2017/09/mmanon-rewrite-finished-for-time-being.html

@t2d

This comment has been minimized.

Show comment
Hide comment
@t2d

t2d Sep 14, 2017

Can we maybe stall the payout of the bounty? I don't have the capacity to extensively test a self-compiled version at the moment. There has been no release or build and I can't check, that it works as expected. But also, I don't want reject the bounty claim.

t2d commented Sep 14, 2017

Can we maybe stall the payout of the bounty? I don't have the capacity to extensively test a self-compiled version at the moment. There has been no release or build and I can't check, that it works as expected. But also, I don't want reject the bounty claim.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Sep 14, 2017

Member

Just FYI: there are daily build packages at least for Ubuntu.

Member

rgerhards commented Sep 14, 2017

Just FYI: there are daily build packages at least for Ubuntu.

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Sep 14, 2017

Member

I should have provided the link... ;-) http://www.rsyslog.com/downloads/download-daily-build/

Member

rgerhards commented Sep 14, 2017

I should have provided the link... ;-) http://www.rsyslog.com/downloads/download-daily-build/

@t2d

This comment has been minimized.

Show comment
Hide comment
@t2d

t2d Sep 14, 2017

Thanks, but I need to test on Debian stable.

t2d commented Sep 14, 2017

Thanks, but I need to test on Debian stable.

@t2d

This comment has been minimized.

Show comment
Hide comment
@t2d

t2d Sep 14, 2017

The latest version for Ubuntu zesty seems to be outdated and not installable

➜  ~ apt-cache madison rsyslog
   rsyslog | 8.26.0~20170322151113-0adiscon1zesty | http://ppa.launchpad.net/adiscon/v8-devel/ubuntu zesty/main amd64 Packages
   rsyslog | 8.16.0-1ubuntu5 | http://de.archive.ubuntu.com/ubuntu zesty/main amd64 Packages
➜  ~ sudo apt install rsyslog
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 rsyslog : Depends: liblogging-stdlog1 but it is not installable
E: Unable to correct problems, you have held broken packages.

t2d commented Sep 14, 2017

The latest version for Ubuntu zesty seems to be outdated and not installable

➜  ~ apt-cache madison rsyslog
   rsyslog | 8.26.0~20170322151113-0adiscon1zesty | http://ppa.launchpad.net/adiscon/v8-devel/ubuntu zesty/main amd64 Packages
   rsyslog | 8.16.0-1ubuntu5 | http://de.archive.ubuntu.com/ubuntu zesty/main amd64 Packages
➜  ~ sudo apt install rsyslog
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 rsyslog : Depends: liblogging-stdlog1 but it is not installable
E: Unable to correct problems, you have held broken packages.
@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Sep 14, 2017

Member

The latest version for Ubuntu zesty seems to be outdated and not installable

I would appreciate if you could open a bug report at https://github.com/rsyslog/rsyslog-pkg-ubuntu as this does not directly relate to the rsyslog project (and the packagers look here only very occasionally). I've sent @friedl a note in any case, but ...

Member

rgerhards commented Sep 14, 2017

The latest version for Ubuntu zesty seems to be outdated and not installable

I would appreciate if you could open a bug report at https://github.com/rsyslog/rsyslog-pkg-ubuntu as this does not directly relate to the rsyslog project (and the packagers look here only very occasionally). I've sent @friedl a note in any case, but ...

@rgerhards

This comment has been minimized.

Show comment
Hide comment
@rgerhards

rgerhards Sep 14, 2017

Member

@mbiebl do you have any idea when rsyslog 8.30 (release target mid october) will make it into debian stable?

Member

rgerhards commented Sep 14, 2017

@mbiebl do you have any idea when rsyslog 8.30 (release target mid october) will make it into debian stable?

@t2d t2d referenced this issue Sep 14, 2017

Closed

Zesty build outdated #64

@mejo-

This comment has been minimized.

Show comment
Hide comment
@mejo-

mejo- Sep 14, 2017

@rgerhards: probably rsyslog 8.30 will never make it into Debian Stretch (current stable). But once rsyslog 8.30 is in Debian Unstable (which usually happens quite fast thanks to the awesome work of @mbiebl), backporting it to stretch-backports should be an easy task.

mejo- commented Sep 14, 2017

@rgerhards: probably rsyslog 8.30 will never make it into Debian Stretch (current stable). But once rsyslog 8.30 is in Debian Unstable (which usually happens quite fast thanks to the awesome work of @mbiebl), backporting it to stretch-backports should be an easy task.

@mbiebl

This comment has been minimized.

Show comment
Hide comment
@mbiebl

mbiebl Sep 14, 2017

Contributor

Atm a stretch backport is rather straightforward so I'm happy to provide one if there is user demand (assuming 8.30.0 doesn't radically change any build dependencies).
@mejo- if you want to see such a 8.30.0 backport, please file a corresponding bug against the Debian package, so I don't forget. Thanks!

Contributor

mbiebl commented Sep 14, 2017

Atm a stretch backport is rather straightforward so I'm happy to provide one if there is user demand (assuming 8.30.0 doesn't radically change any build dependencies).
@mejo- if you want to see such a 8.30.0 backport, please file a corresponding bug against the Debian package, so I don't forget. Thanks!

@jgerhards

This comment has been minimized.

Show comment
Hide comment
@jgerhards

jgerhards Oct 12, 2017

Contributor

FYI: http://jan.gerhards.net/2017/10/slfa-release.html
might be interesting in regards to anonymizing already existing log files.

Contributor

jgerhards commented Oct 12, 2017

FYI: http://jan.gerhards.net/2017/10/slfa-release.html
might be interesting in regards to anonymizing already existing log files.

@t2d

This comment has been minimized.

Show comment
Hide comment
@t2d

t2d Oct 12, 2017

Are you aware of loganon? It seems like a bit of a similar project.

t2d commented Oct 12, 2017

Are you aware of loganon? It seems like a bit of a similar project.

@jgerhards

This comment has been minimized.

Show comment
Hide comment
@jgerhards

jgerhards Oct 13, 2017

Contributor

I've briefly looked at it and decided to start slfa, because it has a much broader focus. For example, one goal is to be able to train analyzers or liblognorm with psodomized output.

Of course, slfa is still in its infancy, but there are already some differences i'd like to point out:

  • slfa is capable of pseudonymizing IP addresses, which might be useful when analyzing log files
  • since loganon is written in python, slfa might be faster (no benchmark yet)
  • in loganon, a regex can only be replaced by a static string, but not randomized, which is possible with slfa

I also plan on implementing more options and parsers to slfa.

Contributor

jgerhards commented Oct 13, 2017

I've briefly looked at it and decided to start slfa, because it has a much broader focus. For example, one goal is to be able to train analyzers or liblognorm with psodomized output.

Of course, slfa is still in its infancy, but there are already some differences i'd like to point out:

  • slfa is capable of pseudonymizing IP addresses, which might be useful when analyzing log files
  • since loganon is written in python, slfa might be faster (no benchmark yet)
  • in loganon, a regex can only be replaced by a static string, but not randomized, which is possible with slfa

I also plan on implementing more options and parsers to slfa.

@jgerhards

This comment has been minimized.

Show comment
Hide comment
@jgerhards

jgerhards Nov 1, 2017

Contributor

@t2d have you tested it yet?

Contributor

jgerhards commented Nov 1, 2017

@t2d have you tested it yet?

@mejo-

This comment has been minimized.

Show comment
Hide comment
@mejo-

mejo- Dec 3, 2017

Hey. Thanks to @mbiebl rsyslog 8.30 is in Debian stretch-backports now. I just gave it a try and it works like a charm for me with the following settings:

module(load="mmanon")
action(type="mmanon" mode="zero" ipv4.bits="32" ipv6.bits="128")

Thanks a lot @jgerhards for implementing it!

mejo- commented Dec 3, 2017

Hey. Thanks to @mbiebl rsyslog 8.30 is in Debian stretch-backports now. I just gave it a try and it works like a charm for me with the following settings:

module(load="mmanon")
action(type="mmanon" mode="zero" ipv4.bits="32" ipv6.bits="128")

Thanks a lot @jgerhards for implementing it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment