New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Whitespace missing from mmdblookup data #1650

Open
GUI opened this Issue Jul 7, 2017 · 2 comments

Comments

Projects
None yet
3 participants
@GUI

GUI commented Jul 7, 2017

When trying the mmdblookup plugin in rsyslog 8.28.0, it looks like whitespace is being stripped from the MaxMind data. For example, if I try the 52.52.118.192 IP address, the city reported from MaxMind's mmdblookup command line utility is San Jose, but the city logged by rsyslog is SanJose (no space).

Let me know if any further configuration details would help.

Here's what I'm seeing in rsyslog's debug logs (note that the spaces seem to be gone from the city data at this point, and possibly related is that some of the data is getting logged twice--the country is logged without spaces as JSON, but with spaces as MaxMind's raw output):

5625.944461324:main Q:Reg/w0  : maxmindb returns: '{"city":{"geoname_id":5392171,"names":{"de":"SanJosé","en":"SanJose","es":"SanJosé","fr":"SanJosé","ja":"サンノゼ","pt-BR":"SanJosé","ru":"Сан-Хосе",},},"continent":{"code":"NA","geoname_id":6255149,"names":{"de":"Nordamerika","en":"NorthAmerica","es":"Norteamérica","fr":"AmériqueduNord","ja":"北アメリカ","pt-BR":"AméricadoNorte","ru":"СевернаяАмерика","zh-CN":"北美洲",},},"country":{"geoname_id":6252001,"iso_code":"US","names":{"de":"USA","en":"UnitedStates","es":"EstadosUnidos","fr":"États-Unis","ja":"アメリカ合衆国","pt-BR":"EstadosUnidos","ru":"США","zh-CN":"美国",},},"location":{"accuracy_radius":1000,"latitude":37.338800,"longitude":-121.891400,"metro_code":807,"time_zone":"America/Los_Angeles",},"postal":{"code":"95141",},"registered_country":{"geoname_id":6252001,"iso_code":"US","names":{"de":"USA","en":"UnitedStates","es":"EstadosUnidos","fr":"États-Unis","ja":"アメリカ合衆国","pt-BR":"EstadosUnidos","ru":"США","zh-CN":"美国",},},"subdivisions":[{"geoname_id":5332921,"iso_code":"CA","names":{"de":"Kalifornien","en":"California","es":"California","fr":"Californie","ja":"カリフォルニア州","pt-BR":"Califórnia","ru":"Калифорния","zh-CN":"加利福尼亚州",},},]}, {
        "geoname_id": 
          6252001 <uint32>
        "iso_code": 
          "US" <utf8_string>
        "names": 
          {
            "de": 
              "USA" <utf8_string>
            "en": 
              "United States" <utf8_string>
            "es": 
              "Estados Unidos" <utf8_string>
            "fr": 
              "États-Unis" <utf8_string>
            "ja": 
              "アメリカ合衆国" <utf8_string>
            "pt-BR": 
              "Estados Unidos" <utf8_string>
            "ru": 
              "США" <utf8_string>
            "zh-CN": 
              "美国" <utf8_string>
          }
      }
    "location": 
      {
        "accuracy_radius": 
          1000 <uint16>
        "latitude": 
          37.338800 <double>
        "longitude": 
          -121.891400 <double>
        "metro_code": 
          807 <uint16>
        "time_zone": 
          "America/Los_Angeles" <utf8_string>
      }
    "postal": 
      {
        "code": 
          "95141" <utf8_string>
      }
    "registered_country": 
      {
        "geoname_id": 
          6252001 <uint32>
        "iso_code": 
          "US" <utf8_string>
        "names": 
          {
            "de": 
              "USA" <utf8_string>
            "en": 
              "United States" <utf8_string>
            "es": 
              "Estados Unidos" <utf8_string>
            "fr": 
              "États-Unis" <utf8_string>
            "ja": 
              "アメリカ合衆国" <utf8_string>
            "pt-BR": 
              "Estados Unidos" <utf8_string>
            "ru": 
              "США" <utf8_string>
            "zh-CN": 
              "美国" <utf8_string>
          }
      }
    "subdivisions": 
      [
        {
          "geoname_id": 
            5332921 <uint32>
          "iso_code": 
            "CA" <utf8_string>
          "names": 
            {
              "de": 
                "Kalifornien" <utf8_string>
              "en": 
                "California" <utf8_string>
              "es": 
                "California" <utf8_string>
              "fr": 
                "Californie" <utf8_string>
              "ja": 
                "カリフォルニア州" <utf8_string>
              "pt-BR": 
                "Califórnia" <utf8_string>
              "ru": 
                "Калифорния" <utf8_string>
              "zh-CN": 
                "加利福尼亚州" <utf8_string>
            }
        }
      ]
  }
'

And compared to what I get when running MaxMind's own mmdblookup command line utility for the same IP:

  {
    "city": 
      {
        "geoname_id": 
          5392171 <uint32>
        "names": 
          {
            "de": 
              "San José" <utf8_string>
            "en": 
              "San Jose" <utf8_string>
            "es": 
              "San José" <utf8_string>
            "fr": 
              "San José" <utf8_string>
            "ja": 
              "サンノゼ" <utf8_string>
            "pt-BR": 
              "San José" <utf8_string>
            "ru": 
              "Сан-Хосе" <utf8_string>
          }
      }
    "continent": 
      {
        "code": 
          "NA" <utf8_string>
        "geoname_id": 
          6255149 <uint32>
        "names": 
          {
            "de": 
              "Nordamerika" <utf8_string>
            "en": 
              "North America" <utf8_string>
            "es": 
              "Norteamérica" <utf8_string>
            "fr": 
              "Amérique du Nord" <utf8_string>
            "ja": 
              "北アメリカ" <utf8_string>
            "pt-BR": 
              "América do Norte" <utf8_string>
            "ru": 
              "Северная Америка" <utf8_string>
            "zh-CN": 
              "北美洲" <utf8_string>
          }
      }
    "country": 
      {
        "geoname_id": 
          6252001 <uint32>
        "iso_code": 
          "US" <utf8_string>
        "names": 
          {
            "de": 
              "USA" <utf8_string>
            "en": 
              "United States" <utf8_string>
            "es": 
              "Estados Unidos" <utf8_string>
            "fr": 
              "États-Unis" <utf8_string>
            "ja": 
              "アメリカ合衆国" <utf8_string>
            "pt-BR": 
              "Estados Unidos" <utf8_string>
            "ru": 
              "США" <utf8_string>
            "zh-CN": 
              "美国" <utf8_string>
          }
      }
    "location": 
      {
        "accuracy_radius": 
          1000 <uint16>
        "latitude": 
          37.338800 <double>
        "longitude": 
          -121.891400 <double>
        "metro_code": 
          807 <uint16>
        "time_zone": 
          "America/Los_Angeles" <utf8_string>
      }
    "postal": 
      {
        "code": 
          "95141" <utf8_string>
      }
    "registered_country": 
      {
        "geoname_id": 
          6252001 <uint32>
        "iso_code": 
          "US" <utf8_string>
        "names": 
          {
            "de": 
              "USA" <utf8_string>
            "en": 
              "United States" <utf8_string>
            "es": 
              "Estados Unidos" <utf8_string>
            "fr": 
              "États-Unis" <utf8_string>
            "ja": 
              "アメリカ合衆国" <utf8_string>
            "pt-BR": 
              "Estados Unidos" <utf8_string>
            "ru": 
              "США" <utf8_string>
            "zh-CN": 
              "美国" <utf8_string>
          }
      }
    "subdivisions": 
      [
        {
          "geoname_id": 
            5332921 <uint32>
          "iso_code": 
            "CA" <utf8_string>
          "names": 
            {
              "de": 
                "Kalifornien" <utf8_string>
              "en": 
                "California" <utf8_string>
              "es": 
                "California" <utf8_string>
              "fr": 
                "Californie" <utf8_string>
              "ja": 
                "カリフォルニア州" <utf8_string>
              "pt-BR": 
                "Califórnia" <utf8_string>
              "ru": 
                "Калифорния" <utf8_string>
              "zh-CN": 
                "加利福尼亚州" <utf8_string>
            }
        }
      ]
  }
@rgerhards

This comment has been minimized.

Member

rgerhards commented Nov 25, 2017

Thanks for opening this issue. Unfortunately, I was hyper-busy with many other things for quite some month and not all issues were properly followed up. I am currently triaging everything.

Unfortunately, there is quite a backlog for the same reason, so I cannot turn to the issue immediately.

@dharrop

This comment has been minimized.

dharrop commented Mar 29, 2018

This issue still seems to be in rsyslog-8.31.0.

Also would like to note that accessing json elements via $!record1!record2 leaves a gap in being able to access the data in "subdivisions" as returned by maxminddb since it's a json group. Perhaps this deserves its own bug report for tracking?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment