No results found on Dashboard, just 5 fields in Index Pattern

[HardcoreHacker]

Hello,

I'm trying to get VulnWhisperer to work but I'm having huge problems with understanding how this should work in ELK. The documentation is very cryptic about this and I'm stuck for days now.

After setting it all up (VulnWhisperer, ELK (ElasticStack 5.x), tested, approved!) I enter Kibana which has no clue what's it's doing on my system, there's no indice or index pattern.

After some digging around I found posts about curling the template to ES to create the index:
curl -XPUT 'http://[IP]:9200/logstash-vulnwhisperer-template' -d@/VulnWhisperer/elasticsearch/logstash-vulnwhisperer-template.json

I've done this, and it says it worked but nothing shows up until I go to "Index Patterns" in Kibana, press "Create New" , then "logstash-vulnwhisperer-*" shows up, I click that, and it goes all wrong it seems, there is no way it can map a "Time Filter" to that Index.

Then I read somewhere else that you must import the dashboards so "Index Patterns" are created, I try this, it says: "the Index Patterns do not exist, Please select the index patterns you'd like re-associated them with." In the field "New index pattern", it does recognize "logstash-vulnwhisperer-*". I click "confirm all changes". Nothing shows up in the "Index Patterns" where it should have added 21 fields.

Any help would be greatly appreciated!

Thanks in advance.

[qmontal]

Hi @HardcoreHacker,

Then I read somewhere else that you must import the dashboards so "Index Patterns" are created

Dashboards won't work without the Index Patterns created. Have you checked the VulnWhisperer's wiki FAQ to see if one of the cases applies to yours? https://github.com/HASecuritySolutions/VulnWhisperer/wiki

After some digging around I found posts about curling the template to ES to create the index:
curl -XPUT 'http://[IP]:9200/logstash-vulnwhisperer-template' -d@/VulnWhisperer/elasticsearch/logstash-vulnwhisperer-template.json

I have never done this, but when I deploy it, I check that ElasticSearch is up doing curl to it, and I check that Logstash is actually detecting the results and processing them into ElasticSearch. If this is working and you have seen Logstash processing the data, it should appear in Kibana as the pattern would be automatically created.

As in the home page is explained, you can try a specific Logstash file to make sure it is working:
sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/1000_nessus_process_file.conf

Cheers!

[HardcoreHacker]

Hello qmontal,

Thank you very very much for your reply!

Yes I have checked the FAQ! But I don't understand how the Index is being build, or how I should configure the time. You say it's automatically created so maybe I should not focus on this now.

I managed to get it to pickup a scan result from the OVA(OpenVAS VM), it all goes OK:
INFO:vulnWhispererOpenVAS:Processing complete

ElasticSearch does work on 9200, it replies on curl as we expect it to.

However, LogStash doesn't reply on 9600 ... OMG I think ... I did setup wrong.

I'll get back to you ASAP!

Thanks a ton!

[qmontal]

It is possible that Logstash doesn't reply on 9600, I also encountered that but it didn't affect the process. While Logstash finds the scanner files and processes/submits them on ElasticSearch (when running the command I mentioned, you should see all the processing going on on the terminal), that means it is actually sending the data to ElasticSearch.

There are a lot of different errors that I have encountered while doing the testing, so we would need a bit more of information in order to find what exactly the issues is ^^ Also, I have personally not dealt with the OpenVAS module, so I am just hoping for not being any issue with that.

[HardcoreHacker]

Hello qmontal,

Thank you very much for your reply again 👍

About logstash not replying, yes I've read that but in my test case it should return, and it does now! I changed 'localhost' to my static IP in the logstash.yml file. I tried first with curl localhost:9600, no dice, again now after stopping/starting logstash and altering the config, I now can curl IP:9600 !

About the processing of files to logstash, you mean the log that says INFO: ? That one seems to return a report just OK, a JSON files is generated at the expected directory as well, but I need to check if the data really goes into ElasticSearch.

I will try for another hour or so, and continue monday if failed :-)

Thanks a lot for your support, very much appreciated !

[qmontal]

When I mention about the logstas processing files, I mean you should see something like this on the command line https://twitter.com/VulnWhisperer/status/1014927125604917250 but being processed at full speed, as it would be all of the records yet to be processed by logstash.

That of course when you execute it direcly on the terminal or with the docker-compose; if it is run as a daemon, no output will be shown, but its a good way to check. Hope you get it to work!

[HardcoreHacker]

Hello qmontal,

I managed to import 3 reports, seems to go OK, it creates the JSON files of those scan results. But when I go to Kibana:

"Unable to fetch mapping. Do you have indices matching the pattern?"

I still am puzzled about those Indexes/Indices and what to do with them to be honest. I've setup a few other instances of ELK before but this is unknown to me.

Because if I don't curl that template (curl -XPUT 'http://[IP]:9200/logstash-vulnwhisperer-template' -d@/VulnWhisperer/elasticsearch/logstash-vulnwhisperer-template.json) I can't even create a new "index pattern" for logstash-vulnwhisperer-*

It currently seems like:

• ES is OK (services running, curl replies)
• Logstash is OK (services running, curl replies)
• Kibana is OK (services running, curl replies)
• VulnWhisperer is OK (config, can connect to my OpenVAS box

After reading your last comment, I tried:
/usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/3000_openvas.conf

it was very busy for ~30secs, then it only said:
"Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties"

I can only find instances of this log4j2.properties file in /etc/logstash and /etc/elasticsearch I don't think this requires any configuration.

I've started to inspect the /var/log/logstash dir and it says ES instance is dead, it's still on localhost it seems, alright, trying to fix that :-)

Thanks again!

[qmontal]

Regarding the Unable to fetch mapping. Do you have indices matching the pattern?, this was at the FAQ pointing to this other issue #83 (comment), but I believe it is pretty this has been already covered by your troubleshooting. If you can take another look just to make sure to completely discard it, would be great ^^

[HardcoreHacker]

Hello qmontal,

I read that particular comment, thanks for mentioning it as it did shine some light.

I'll be continuing this in a few days, maybe I will start over from scratch to avoind any other config error, as I suspect I did setup something wrong.

Thanks for your support, and have a great weekend!

[HardcoreHacker]

Well, now I ELK running properly, all seems OK, I see a lot of logging when I now import from the OpenVAS OVA:
[INFO ]logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
.... New elasticsearch output ....
[INFO ][logstash.agent] Successfully started Logstsh API endpoint {:port=>9600}

But Kibana is still 100% unconfigured etc, there is no Index Pattern created. So I CURL the template, make a "logstash-vulnwhisperer-*" Index Pattern (still major problems with the time fields), then I can import the Dashboards into Kibana (without Index Pattern, it's unable to import anything), and after importing all the Dashboards, the fields of the Index Pattern are still stuck on just the 5 defaults, _id, _index, _score, _source, _type, where we should have at least 21 more if I understand it correctly.

I'm still stuck on how Kibana even could work, I don't understand how the Index Patterns should be created (in this case), and it's obviously now failing on only the Kibana part as we see LogStash/ES respond in the logging as we expect.cat

When I start logtash manually as you describe:

/usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/3000_openvas.conf

I see the data being processed, it only gives one error:

_main ERROR Unable to locate appender "${sys:ls.log.format}rolling" for logger config "root"
Sending LogStash's logs to /var/log/logstash which is now configured via log4j2.properties

I'm still totally lost what to do with Kibana here, I want to set it up:

• Index Pattern (how for VulnWhisperer can we setup a pattern? the curl / template doesn't seem to do the trick as it's missing the proper timefields)
• Indices
• Dashboards (should be just importing, but without a Index Pattern, that's not happening)

Any help on this would again be greatly appreciated!

I can BTW see more people having this problem:
https://discuss.elastic.co/t/logstash-is-creating-a-directory-literally-called-sys-ls-logs/100126/4
(running as user logstash ( sudo -Hu logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash/) or normal user doesn't resolve the error)
I'm running Debian, perhaps it somehow fails to translate ${sys:ls.logs}?

Sorry for the long post, tl;dr:

• Still figuring out how Kibana works with VulnWhisperer
• The ${sys:ls.log} problem is unrelated to VulnWhisperer and applies to LogStash, I read that only people who apt-get the package get this problem, still figuring it out.

Next I'll try to:

• Check permissions of the path.settings of logstash

But for now I'll have to leave it as it is and do something else :-)

[qmontal]

_main ERROR Unable to locate appender "${sys:ls.log.format}rolling" for logger config "root"
Sending LogStash's logs to /var/log/logstash which is now configured via log4j2.properties

Unfortunately I haven't seen that error before.

I understand that you have ELK directly installed in the host, could you try the docker-compose VulnWhisperer file to make sure this is something related to your setup/debian or if it is actually and issue related vulnwhisperer/openvas module? Here are the instructions for that.

Also, maybe if you join the VulnWhisperer slack channel I am able to help you better being more responsive :) You can join the channel here.

[HardcoreHacker]

Hello qmontal,

Docker is not an option for me sadly. But I managed to fix the " main ERROR Unable to locate appender "${sys:ls.log.format}rolling" for logger config "root" " problem:

it was just a permission problem (shown in a unlogical way by log4j?):
rm /var/log/logstash/* (just to remove any file with wrong permissions)
chmod 777 /var/lib/logstash/queue (just to be sure it can write)
chmod 777 /var/log/logstash (this is probably what fixed it)

now it does log properly, no errors anymore!

I'll contact you on the channel, thanks again!

[qmontal]

Hey @HardcoreHacker,

I don't exactly remember what was exactly the issue, but I believe we solved this over Slack? Could we close this ticket?
In case you remember what the issue was, it would be great if you could add some words for the next users ^^

Cheers!

[HardcoreHacker]

Hello Qmontal,

sorry for not getting back to this but I did leave a message on Slack that I couldn't invest more time into getting VulnWhisperer to work. I tried Seccubus and it was working with minor changes. This can be closed, I've managed to fix a lot of issues while installing, all those fixes were shared on Slack.

Thanks for your efforts helping me back then!

With kind regards!

[qmontal]

Hey @HardcoreHacker!

That's true, I forgot you mentioned it :) I hope we can fix the issues with VulnWhisperer so that next time you want to take it a look it works as flawlessly as possible ^^

Thank you for testing our project 😄

[HardcoreHacker]

You're very welcome! And thanks a lot for the support, that's highly appreciated! I also hope VulnWhisperer can manage to install outside Docker more easy, because the idea of VulnWhisperer is very nice (exporting scan results to ELK)

Good luck with the development!

I'll keep an eye on the development +1