New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Couldn't refresh data adapter #4748
Comments
@ceveyp Please post the complete error message including some context from the logs of your Graylog node and provide the complete configuration of the lladdr-lookup data adapter. |
Hi. You said to post the context around the error message but actually there really is no context, the log file is just filled with these types of errors messages one after the other. Here is an example error message: 2018-05-04T22:07:13.195-04:00 ERROR [LookupDataAdapter] Couldn't refresh data adapter lladdr-lookup/5aa40ca39d992d72500b1b83/@753740cb Here is my configuration for lladdr-lookup data adapter: Here is what my hosts.csv file looks like: Also one thing I should say, these lookup tables use to work. Maybe I accidentally updated Java. But they don't work anymore. When I create a new lookup table I get the same problem. |
Tagging on this issue - I encountered this same problem. The lookup tables have been functional in the past, and I didn't notice an issue until I had to restart all of our Graylog servers over the weekend. It appears most of our lookup tables were not working, and there are messages in the log indicating a problem. We have a lookup table that's used many times per second, and I was seeing this message in server.log: "WARN [LookupTableService] Lookup table <banner_hosts_ip_table> does not exist" This table did exist, and initial inspection showed no obvious configuration issues. I did the following:
I updated java, and that seemed to fix the issue for me:
This fixed 2 of our 5 nodes. I am working down the list to see if that clears all of the issues up. |
I seriously can't believe someone would use Graylog in a production environment. Honestly I spent about 3 weeks troubleshooting Graylog and I just gave up on it in the end. It is infinitely more work than it's worth. |
This is open source software in a nutshell: it takes forever to deploy, you have to go right down in to the nitty-gritty technical details, and at the same time the documentation is atrocious, and then after you finally set it up the way you want, you run in to all these weird problems for no fault of your own, with no meaningful way troubleshoot. |
Updating java didn't fix my issue - just took a few minutes to come back. After looking at my issue for a bit, I have a difference compared to what you posted. I see messages warning me that a particular lookup table doesn't exist. "WARN [LookupTableService] Lookup table does not exist" The lookup table does exist in the GUI, and I verified I can see it in mongodb.
Here is a snip of logs showing what looks odd to me:
At this point I believe that none of my lookup tables are working. |
Yeah I reinstalled Graylog, I updated Java, I recreated the lookup table it made no difference. |
I have more information. Restating my current issue: Some of my previously working lookup tables have ceased to function. A review of my server.log file complains that tables do not exist, but I can see them in the gui and in mongodb. I renamed a lookup table and updated the pipelines that consume it, but still get errors in the log. Here are config screenshots for one of the lookup tables with an issue. All of the logs are full of: 2018-07-10T08:36:31.736-04:00 WARN [LookupTableService] Lookup table <banner_hosts_ip_lookuptable> does not exist I attempted a restart of everything to get clean logs before posting. This proved a little challenging - log files were cleared of useful information after 4-5 seconds of actual traffic. While tailing the logs this time, I caught this message:
I understand now that the LookupTableService is probably not stopping/starting properly. Here are the logs from gl03 from when a restart is attempted. |
In #4748 (comment) looks like it fails to refresh the change information for that file. There's code that is supposed to avoid anything being The code in question hasn't changed recently, so something else must be going on. It very much sounds like Graylog being unable to read the file (or update its change status), which appears to happen during the first load. Can you try copying the file to a location where you are absolutely sure it is readable (e.g. Thanks! |
@dustintennill Thank you for the detailed information. According to your log file, the "alexa-top-million" data adapter is starting but never goes into the
The lookup table code is currently waiting for every data adapter and cache to start. That means if there is one data adapter that doesn't go into the "RUNNING" state, no lookup table gets started. This is another issue that we need to look into. But first we should try to get your data adapter running or at least figure out what the problem is. Can you please show us the configuration of your "alexa-top-million" (ID You can get the configuration via curl on the graylog node:
Thank you! |
Here is the output. The CSV file in question is one million lines.
I just realized that this config is for a version of this lookup table we are not using. We had trouble with the DSV setting and decided to go back to local CSV files until we had time to troubleshoot. We created a new data adapter using local CSV files, and that was working fine. However, I did not delete the unused DSV version of the data adapter at that time. To sum up what the issue was and how it caught us: I have deleted the unused adapter and am restarting services. |
Confirmed - removing the DSV table and restarting services resolved the issues we were having. I don't know if this resolves the issue @ceveyp was having. |
@dustintennill Thank you for the update! I guess either loading the CSV file from the HTTP server or loading the file into heap memory takes some time in your case. We need to make this more robust to avoid having one faulty or slow data adapter block all lookup tables. I will open another issue for that. Regarding the NPE @ceveyp reported: I will create a small fix to avoid a potential NullPointerException in the CSV data adapter. If that fixes the issue is hard to say without complete logs, though. |
Make sure the "fileInfo" field is initialized with an empty FileInfo object. Also catch all exceptions in FileInfo#forPath instead of IOExceptions only. Refs #4748
Make sure the "fileInfo" field is initialized with an empty FileInfo object. Also catch all exceptions in FileInfo#forPath instead of IOExceptions only. Refs Graylog2/graylog2-server#4748 (cherry picked from commit Graylog2/graylog2-server@7759690)
) Make sure the "fileInfo" field is initialized with an empty FileInfo object. Also catch all exceptions in FileInfo#forPath instead of IOExceptions only. Refs Graylog2/graylog2-server#4748 (cherry picked from commit Graylog2/graylog2-server@7759690)
Closing this. The NullPointerException has been fixed and will be available in the upcoming 3.0 release. The other issue with the blocking data adapter has been added to #4524. Please watch that issue for progress on that one. Thank you! |
I had weird errors like |
Hello everyone, Are you sure this bug is resolved?
Could you please check it ? |
Expected Behavior
Lookup tables should be updated.
Current Behavior
In the log file I get messages like this:
2018-04-21T16:20:31.544-04:00 ERROR [LookupDataAdapter] Couldn't refresh data adapter lladdr-lookup/5aa40ca39d992d72500b1b83/@7e3d1a53
java.lang.NullPointerException: null
Also when I create a lookup table or insert rows in an existing CSV file, the lookup doesn't work.
Possible Solution
Is it related to my version of Java? I'm just using the current latest version of Java, 1.8.0_162. Is Graylog not compatible with this version of Java?
Context
Your Environment
The text was updated successfully, but these errors were encountered: