Lookup tables should be updated.
In the log file I get messages like this:
2018-04-21T16:20:31.544-04:00 ERROR [LookupDataAdapter] Couldn't refresh data adapter lladdr-lookup/5aa40ca39d992d72500b1b83/@7e3d1a53
Also when I create a lookup table or insert rows in an existing CSV file, the lookup doesn't work.
Is it related to my version of Java? I'm just using the current latest version of Java, 1.8.0_162. Is Graylog not compatible with this version of Java?
The text was updated successfully, but these errors were encountered:
Hi. You said to post the context around the error message but actually there really is no context, the log file is just filled with these types of errors messages one after the other.
Here is an example error message:
2018-05-04T22:07:13.195-04:00 ERROR [LookupDataAdapter] Couldn't refresh data adapter lladdr-lookup/5aa40ca39d992d72500b1b83/@753740cb
Here is what my hosts.csv file looks like:
Also one thing I should say, these lookup tables use to work. Maybe I accidentally updated Java. But they don't work anymore. When I create a new lookup table I get the same problem.
Tagging on this issue - I encountered this same problem.
The lookup tables have been functional in the past, and I didn't notice an issue until I had to restart all of our Graylog servers over the weekend. It appears most of our lookup tables were not working, and there are messages in the log indicating a problem.
We have a lookup table that's used many times per second, and I was seeing this message in server.log: "WARN [LookupTableService] Lookup table <banner_hosts_ip_table> does not exist"
This table did exist, and initial inspection showed no obvious configuration issues.
I did the following:
I updated java, and that seemed to fix the issue for me:
This fixed 2 of our 5 nodes. I am working down the list to see if that clears all of the issues up.
This is open source software in a nutshell: it takes forever to deploy, you have to go right down in to the nitty-gritty technical details, and at the same time the documentation is atrocious, and then after you finally set it up the way you want, you run in to all these weird problems for no fault of your own, with no meaningful way troubleshoot.
Updating java didn't fix my issue - just took a few minutes to come back. After looking at my issue for a bit, I have a difference compared to what you posted. I see messages warning me that a particular lookup table doesn't exist.
"WARN [LookupTableService] Lookup table does not exist"
The lookup table does exist in the GUI, and I verified I can see it in mongodb.
Here is a snip of logs showing what looks odd to me:
At this point I believe that none of my lookup tables are working.
I have more information.
Restating my current issue: Some of my previously working lookup tables have ceased to function. A review of my server.log file complains that tables do not exist, but I can see them in the gui and in mongodb. I renamed a lookup table and updated the pipelines that consume it, but still get errors in the log.
All of the logs are full of: 2018-07-10T08:36:31.736-04:00 WARN [LookupTableService] Lookup table <banner_hosts_ip_lookuptable> does not exist
I attempted a restart of everything to get clean logs before posting. This proved a little challenging - log files were cleared of useful information after 4-5 seconds of actual traffic.
While tailing the logs this time, I caught this message:
I understand now that the LookupTableService is probably not stopping/starting properly.
Here are the logs from gl03 from when a restart is attempted.
In #4748 (comment) looks like it fails to refresh the change information for that file. There's code that is supposed to avoid anything being
The code in question hasn't changed recently, so something else must be going on. It very much sounds like Graylog being unable to read the file (or update its change status), which appears to happen during the first load.
Can you try copying the file to a location where you are absolutely sure it is readable (e.g.
@dustintennill Thank you for the detailed information.
According to your log file, the "alexa-top-million" data adapter is starting but never goes into the
The lookup table code is currently waiting for every data adapter and cache to start. That means if there is one data adapter that doesn't go into the "RUNNING" state, no lookup table gets started. This is another issue that we need to look into.
But first we should try to get your data adapter running or at least figure out what the problem is. Can you please show us the configuration of your "alexa-top-million" (ID
You can get the configuration via curl on the graylog node:
Here is the output.
The CSV file in question is one million lines.
I just realized that this config is for a version of this lookup table we are not using. We had trouble with the DSV setting and decided to go back to local CSV files until we had time to troubleshoot. We created a new data adapter using local CSV files, and that was working fine. However, I did not delete the unused DSV version of the data adapter at that time.
To sum up what the issue was and how it caught us:
I have deleted the unused adapter and am restarting services.
@dustintennill Thank you for the update!
I guess either loading the CSV file from the HTTP server or loading the file into heap memory takes some time in your case. We need to make this more robust to avoid having one faulty or slow data adapter block all lookup tables. I will open another issue for that.
Regarding the NPE @ceveyp reported: I will create a small fix to avoid a potential NullPointerException in the CSV data adapter. If that fixes the issue is hard to say without complete logs, though.
Make sure the "fileInfo" field is initialized with an empty FileInfo object. Also catch all exceptions in FileInfo#forPath instead of IOExceptions only. Refs #4748
Are you sure this bug is resolved?
Could you please check it ?