Elasticsearch/Logstash/Kibana 7.6.1

[renini]

Many thanks for your work on this! I'm trying to fix the visualizations and dashboards for kibana 7.6.1.
Did you by any chance look into this already?

[nin9s]

I'm currently running elk 7.6.0 - what issues are you facing with 7.6.1?

[renini]

I had some troubles importing the ndjson with saved objects on 7.6.0, but i just tried again on 7.6.1 and now it seems to work.

I do have a dashboard now, with some errors though:
[esaggs] > Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [date] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-syslog-dns-2020.03","node":"dTwjoMTVTHa3hycvL1AieQ","reason":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [date] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}]

Is elk-hole - vis_and_dash.ndjson the only file to import?

[nin9s]

Is elk-hole - vis_and_dash.ndjson the only file to import?

in recent elk versions, yes.
I'm afraid this has something to do with the changes field types in the recent elk versions. I'm not exactly sure what they did and what I need to change in my template.
Did you import the template successfully?

what do you need to do to raise this error?

its been a while since I last digged really deep into elastic and its mechanic so forgive the lack of clarity

[renini]

Kibana -> Import saved objects -> vis_and_dash.ndjson seems to work just fine:
Successfully imported 15 objects.

But then while opening the DNS - pihole dashboard you get this error:
Alternatively, set fielddata=true on [date] in order to load field data by uninverting the inverted index.
This is on all panels but not on:

• Requests over time - pihole
• Requests vs piholed - pihole
• piholed percent - pihole

https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html
I'm gonna look into it. Just wondering if you experience this yourself too?

[renini]

At step 15 under KIBANA HOST (CAN BE THE SAME AS LOGSTASH AND ELASTICSEARCH)
in the README, you mention:
Click next step and select @Timezone

Mine doesn't show @timezone, but i assume you mean @timestamp?

[nin9s]

Mine doesn't show @Timezone, but i assume you mean @timestamp?

yes, thanks for that!

thanks for your effort, I'll have a look into this but it can take a bit because of the current covid situation and so on.

[klausagnoletti]

I have a bit of problems in ELK 7.6.1 as well. First of all, Kibana says that there are 94 index fields (and not 79). And one of them is not geoip.location so that dashboard fails. However, there are other geoip-fields, such as geoip.location.lat and geoip.location.lon. I have tried rebuilding and re-adding indexes but it doesn't help. Do you have an idea why? Searching the interwebs leads me to somewhat similar issues but nothing that really helps. This could partially be because I don't really know what I am doing :-)

Any help is appreciated.

Thanks

/klaus

[nin9s]

I'm going to export my live dashboard and running template because I don't have these issues myself. Am currently away so please ping me here if I forget about it

[andrema2]

Hi, I have the same issue. I was having this issue with the pfSense too. What I saw is that the GeoIP data is coming from /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/

How can I point it to somewhere else ?

[nin9s]

@klausagnoletti As I thought a little bit more bout it I remembered I sometimes had the geoip.location field problem too. So now I just need to remember how I fixed that :>

@klausagnoletti & @andrema2 I found that with the recent version there 83 fields here so the howto install part should be updated because the number of fields may vary depending on the used elk version. The discrepancy should not be a problem in general.

Could you please paste your current index template:

GET /_template/logstash-syslog-dns

{
  "logstash-syslog-dns" : {
    "order" : 0,
    "index_patterns" : [
      "logstash-syslog-dns*"
    ],
    "settings" : { },
    "mappings" : {
      "dynamic" : "true",
      "properties" : {
        "date" : {
          "format" : "MMM  d HH:mm:ss||MMM dd HH:mm:ss",
          "type" : "date"
        },
        "agent" : {
          "properties" : {
            "hostname" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "name" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "id" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "ephemeral_id" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "type" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "version" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "norms" : false,
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "ignore_above" : 256,
                      "type" : "keyword"
                    }
                  }
                }
              }
            },
            "offset" : {
              "type" : "long"
            }
          }
        },
        "blocked_domain" : {
          "norms" : false,
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "pid" : {
          "type" : "integer"
        },
        "program" : {
          "norms" : false,
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "type" : {
          "norms" : false,
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "logrow" : {
          "type" : "integer"
        },
        "ip_response" : {
          "type" : "ip"
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            }
          }
        },
        "source_fqdn" : {
          "norms" : false,
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "source_port" : {
          "type" : "integer"
        },
        "@version" : {
          "type" : "keyword"
        },
        "host" : {
          "properties" : {
            "name" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            }
          }
        },
        "ip_request" : {
          "type" : "ip"
        },
        "dns_forward_to" : {
          "type" : "ip",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "geoip" : {
          "dynamic" : "true",
          "properties" : {
            "timezone" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "continent_code" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "city_name" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "country_code2" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "country_name" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "dma_code" : {
              "type" : "long"
            },
            "country_code3" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "location" : {
              "type" : "geo_point"
            },
            "region_name" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "postal_code" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "longitude" : {
              "type" : "half_float"
            },
            "region_code" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            }
          }
        },
        "source_host" : {
          "type" : "ip"
        },
        "query_type" : {
          "norms" : false,
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "message" : {
          "norms" : false,
          "type" : "text"
        },
        "domain_response" : {
          "norms" : false,
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "pihole" : {
          "type" : "ip"
        },
        "tags" : {
          "type" : "keyword",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "input" : {
          "properties" : {
            "type" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            }
          }
        },
        "request_from" : {
          "type" : "ip"
        },
        "@timestamp" : {
          "type" : "date"
        },
        "domain_request" : {
          "norms" : false,
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        }
      }
    },
    "aliases" : { }
  }
}

[headbug]

I also am having this issue, with geoip.location missing, my output is

{
  "logstash-syslog-dns": {
    "order": 0,
    "index_patterns": [
      "logstash-syslog-dns*"
    ],
    "settings": {},
    "mappings": {
      "dynamic": "true",
      "properties": {
        "date": {
          "format": "MMM  d HH:mm:ss||MMM dd HH:mm:ss",
          "type": "date"
        },
        "agent": {
          "properties": {
            "hostname": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "name": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "id": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "ephemeral_id": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "type": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "version": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "norms": false,
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "ignore_above": 256,
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "offset": {
              "type": "long"
            }
          }
        },
        "blocked_domain": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "pid": {
          "type": "integer"
        },
        "program": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "type": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "logrow": {
          "type": "integer"
        },
        "ip_response": {
          "type": "ip"
        },
        "ecs": {
          "properties": {
            "version": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            }
          }
        },
        "source_fqdn": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "source_port": {
          "type": "integer"
        },
        "@version": {
          "type": "keyword"
        },
        "host": {
          "properties": {
            "name": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            }
          }
        },
        "ip_request": {
          "type": "ip"
        },
        "dns_forward_to": {
          "type": "ip",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "geoip": {
          "dynamic": "true",
          "properties": {
            "timezone": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "ip": {
              "type": "ip"
            },
            "latitude": {
              "type": "half_float"
            },
            "continent_code": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "city_name": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "country_code2": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "country_name": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "dma_code": {
              "type": "long"
            },
            "country_code3": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "location": {
              "type": "geo_point"
            },
            "region_name": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "postal_code": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "longitude": {
              "type": "half_float"
            },
            "region_code": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            }
          }
        },
        "source_host": {
          "type": "ip"
        },
        "query_type": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "message": {
          "norms": false,
          "type": "text"
        },
        "domain_response": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "pihole": {
          "type": "ip"
        },
        "tags": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "input": {
          "properties": {
            "type": {
              "norms": false,
              "type": "text",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            }
          }
        },
        "request_from": {
          "type": "ip"
        },
        "@timestamp": {
          "type": "date"
        },
        "domain_request": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      }
    },
    "aliases": {}
  }
}

[nin9s]

@headbug interesting. did you refresh the index fields? if you filter for "geoip" what fields do you have?

[nin9s]

please note: you MUST re-index after applying the index template (meaning the data in the current index will be lost). Otherwise the new index template will not applied

[andrema2]

How do I do that ?

[nin9s]

there are a couple of ways possible.
easiest solution would be via kibana

[headbug]

Hah, my kibana had migrating indexes conflict, so had to delete .kibana_1, now everything is gone. Will start over.

[andrema2]

After I delete everything looks fine. At the same time I upgraded to 7.7.0

[nin9s]

After I delete everything looks fine. At the same time I upgraded to 7.7.0

So for you issue solved?

[andrema2]

Yes, it is

[headbug]

Ok, I'm back. Still no geoip.location though.

On a side note, a comment about setting timezones in 20-dns-syslog.conf would have spared me a couple of hours of debugging :)

Like so:

date {
    match => [ "date", "MMM  d HH:mm:ss","MMM dd HH:mm:ss" ]
    timezone => "CET"
  }

[nin9s]

Ok, I'm back. Still no geoip.location though.

logstash* isnt the correct index pattern name and therefore the geoip mapping in logstash #31 (comment)* is not included.
have a look at the correct index pattern

On a side note, a comment about setting timezones in 20-dns-syslog.conf would have spared me a couple of hours of debugging :)

indeed. fun fact is I'm located in vienna so I need CET too 👍

[andrema2]

This is mine template.

[nin9s]

This is mine template.

To be clear, it's the index pattern not the template, but yes it's the correct one

The index template gets applied to the index pattern

[andrema2]

I got it from the index template page. Isn't it the right place ?

[nin9s]

I got it from the index template page. Isn't it the right place ?

It is, don't worry. Just realised I wanted to correct you about template/pattern and saw I used the incorrect term a few posts above too.

[headbug]

Same same, even with the right index pattern name

[nin9s]

Did you reindex?

[headbug]

No, I hadn't. And now:

Thank you very much!

[nin9s]

@klausagnoletti your issue remains?

[headbug]

Is there any way I can trace the originating machine on my local net, using elk-hole? I only have source_host set to my GW, for all traffic

[nin9s]

do you use your gateway as local dns resolver and your gateway is forwarding all requests to pihole? If so, there is no way of extracting the real source ip

[headbug]

Yep, exactly so. Thanks

[nin9s]

So why don't you advertise the pihole address to your clients?

[headbug]

I could, but the ease of having to manage only one entry is really nice. I have ca 20 devices/computers/servers to manage and setting dns manually is a chore. Also I want to be able to trace suspicious behavior if I let an unknown device on my network. I've so far seen traffic to china on addresses tied to malware which I suspect are coming from my wife's new laptop from work.

[nin9s]

To be honest this does not make sense to me. For your dhcp clients you can distribute the pihole address easily. I'm having more than 20 mashines here and it would be a matter of minutes to change the dns for all. For all servers/mashines with static IPs you only need to change that one time.
And of course you'll than get the benefit of being able to trace suspicious clients because you can actually see the clients ip address.
Changing the resolver IP for alle clients seems to fix all issues you are currently facing, and it should be faster than using your gateway, dont you think?

[headbug]

Ok, that might be a gap in my home networking knowledge, but I have googled and looked in all settings I can think of, but for my life I cannot see how I would push my pi-hole IP as dns to all DHCP clients, they all have the GW-address as dns, and in the GW settings I have put in the pi-hole address. How would I go about to let the GW publish the pi-hole address to all connected DHCP clients?

[nin9s]

Depends on the actual Gateway if it's possible or not. Sometimes ISPs lock this setting for the sake of simplicity. What device do you use?
Alternatively disable the dhcp at the gateway and use your own dhcp server

[headbug]

I've put my own router, an Ubiquity Amplify after my isps modem and now found a setting: "Bypass dns cache, use upstream dns". Is that the one?

…

On Mon, May 18, 2020 at 1:31 PM 9S ***@***.***> wrote: Depends on the actual Gateway if it's possible or not. Sometimes ISPs lock this setting for the sake of simplicity. What device do you use? Alternatively disable the dhcp at the gateway and use your own dhcp server — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#31 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGGAS7P7VREEO77W7JDH2U3RSEMAXANCNFSM4LI7BE6A> .

[headbug]

It is the one! Thanks!