New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dashboard widget does not update data #2301

Closed
OlympiaLady opened this Issue May 27, 2016 · 29 comments

Comments

Projects
None yet
6 participants
@OlympiaLady

OlympiaLady commented May 27, 2016

I created a dashboard and added a widget that is supposed to report the number of IIS log entries for my servers within the last two hours. It does not seem to update. When I click on the "Replay search" button and view the search results, the data is different (and accurate I believe). The query is simple.

SourceName:IIS
Time: Last 2 hours

Steps to reproduce:

  1. Collect some IIS log data.
  2. Search the last two hours of data by using the Elasticsearch query SourceName:IIS.
  3. Drill down on the Source field and click on Quick values.
  4. Click the Add to dashboard pull down box and add it to my Beth's dashboard.
  5. Choose show pie chart and data table.
  6. View the dashboard and see the new widget.
  7. You see other widgets updating their data.
  8. This widget does not update.
  9. Close your browser windows and re-open. Same problem.
  10. Try a different browser (Edge), see the same problem.

Environment

  • Graylog Version: 2.0.1 appliance
  • Elasticsearch Version: 2.3.1
  • MongoDB Version: 3.2.5
  • Operating System: Ubuntu 14.04
  • Browser version: Chrome 50.0.2661.102 (primary), Edge 25.10586.0.0 (secondary testing)

image

old_data

@kroepke

This comment has been minimized.

Member

kroepke commented May 30, 2016

Two questions:

  1. Is "update in background" enabled if the browser isn't focussed?
  2. What is the cache time of that widget? Widget values are cached for that amount of time and will not be reloaded before that time is elapsed.

Thanks!

@tachtler

This comment has been minimized.

tachtler commented May 31, 2016

Hi,

I have a similar problem. I defined a dashboard with a widget as follows:

Title: All messages
Cache time: 10
Time range type: Keyword
Search keyword: midnight yesterday to midnight today
Search query: [EMPTY]
Display trend: yes
Lower is better: yes

After login to graylog and click on the Dashbord, no data will be updated, onyl when I
1. Unlock / Edit
2. Edit the widget
3. Update-Button

Then the data will be updated.

This is problem occurs since version 2.0.x.

Thank you for your Help!
Klaus.

@joschi

This comment has been minimized.

Contributor

joschi commented May 31, 2016

@tachtler Which web browser are you using (exact version and operating system)? Do the dashboard widgets display the correct data at all or just after running the procedure you've described?

@tachtler

This comment has been minimized.

tachtler commented May 31, 2016

Hi joschi,

OS: CentOS 7.2
Browser: Firefox ESR 45.1.1 (from OS)

The correct data will be only display, after the dashboard procedure. I have 5 widget defined, all with the same setup in:

Cache time: 10
Time range type: Keyword
Search keyword: midnight yesterday to midnight today
Search query: [EMPTY]

Thank you!
Klaus.

@joschi joschi added the web label May 31, 2016

@edmundoa

This comment has been minimized.

Member

edmundoa commented May 31, 2016

Hi,

Yet another couple of questions on my side:

  • Does clicking on "Update in the background/foreground" button has any effect on this behaviour?
  • Is there any output logged in your browser's developer console while having the dashboards page opened?

Thank you

@tachtler

This comment has been minimized.

tachtler commented May 31, 2016

Hi Edmundo,

  • Clicking on "Update in the background/foreground" button has NO effect to this behavior
  • There is also NO output logged in the console at the bottom, only when I click on the update button in edit mode, the normal green line was shown.

Thank you!
Klaus.

@sander1133

This comment has been minimized.

sander1133 commented May 31, 2016

Unfortunately same problem here, since upgraded to version 2.0.1 widgets aren't updating anymore.

@OlympiaLady

This comment has been minimized.

OlympiaLady commented May 31, 2016

Hi,

Here are my responses:

Two questions:

  1. Is "update in background" enabled if the browser isn't focussed? - I have tried it both with Update in foreground and Update in Background. No difference.
  2. What is the cache time of that widget? Widget values are cached for that amount of time and will not be reloaded before that time is elapsed. - I have tried different cache times, but my widget now has a cache time of 5 seconds. If I change the cache time, it seems to update the data. It does not update again until I change the cache time, however.

Does clicking on "Update in the background/foreground" button has any effect on this behaviour?
Is there any output logged in your browser's developer console while having the dashboards page opened? - Clicking on the Update in the background/foreground has no effect on the behavior.

Is there any output logged in your browser's developer console while having the dashboards page opened? - I am not a developer, but I installed Web Developer on chrispederick.com and got this output for the pie chart:

@OlympiaLady

This comment has been minimized.

OlympiaLady commented May 31, 2016

2016-05-31 08_28_30-dashboard_widget

@edmundoa

This comment has been minimized.

Member

edmundoa commented May 31, 2016

Sorry folks, but I'm unable to reproduce this issue, I need more data.

  • Do you see any errors on your Graylog server logs while the dashboard is open?
  • Can you see any activity in your browser's developer tools network tab? Here's a link on how to open them in Chrome. I attached a screenshot with what I see, so you can compare. Please ensure you click somewhere inside the dashboard once you open the network tab, in case you didn't enable the dashboard updates in the background.
    screen shot 2016-05-31 at 17 36 57

Additionally, it would be really helpful if one of you could share a widget configuration with us. To do so, you need to use the REST API, for instance opening the API browser usually available in http://<graylog-server-host>:12900/api-browser. Then query the /dashboards/{dashboardId}/widgets/{widgetId} resource and post the results for one of your broken widgets.

@edmundoa

This comment has been minimized.

Member

edmundoa commented May 31, 2016

By the way, we fixed a couple of issues with old dashboard widgets in 2.0.2. They don't look like the problem you are experiencing, but you can try upgrading just to be sure.

@edmundoa edmundoa self-assigned this May 31, 2016

@OlympiaLady

This comment has been minimized.

OlympiaLady commented May 31, 2016

I'll try to get to this today.
Beth

@tachtler

This comment has been minimized.

tachtler commented Jun 1, 2016

Hi,

I query my /dashboards/{dashboardId} and widgets/{widgetId} resource and the results for one of my broken widgets.

{"creator_user_id":"root","description":"Suche von Mitternacht vorgestern bis Mitternacht gestern","created_at":"2016-04-29T07:04:21.809Z","positions":{"24aff977-bbc0-4d46-af00-2d67c7264a36":{"width":1,"col":3,"row":1,"height":1},"7b23de7a-9730-4083-b176-69eba399a7c5":{"width":1,"col":2,"row":1,"height":1},"9fe19cfc-13df-4114-a7b6-265524575cb6":{"width":1,"col":2,"row":2,"height":2},"de7e0cbc-bd93-402f-9586-ef1db016451e":{"width":1,"col":3,"row":2,"height":2},"b1abaef0-7976-4dbc-8958-0ffafae7239c":{"width":1,"col":1,"row":1,"height":3}},"id":"57230775c7df8a3c2186d4cf","title":"24-Stunden Meldungsübersicht","widgets":[{"creator_user_id":"root","cache_time":10,"description":"Problem Meldungen","id":"24aff977-bbc0-4d46-af00-2d67c7264a36","type":"SEARCH_RESULT_COUNT","config":{"timerange":{"type":"keyword","keyword":"midnight yesterday to midnight today"},"lower_is_better":true,"trend":true,"query":"(level:0 OR level:1 OR level:2 OR level:3 OR level:4)"}},{"creator_user_id":"root","cache_time":10,"description":"Meldungen gesamt","id":"9fe19cfc-13df-4114-a7b6-265524575cb6","type":"QUICKVALUES","config":{"timerange":{"type":"keyword","keyword":"midnight yesterday to midnight today"},"field":"level","show_pie_chart":false,"query":"","show_data_table":true}},{"creator_user_id":"root","cache_time":10,"description":"Meldungen gesamt","id":"b1abaef0-7976-4dbc-8958-0ffafae7239c","type":"QUICKVALUES","config":{"timerange":{"type":"keyword","keyword":"midnight yesterday to midnight today"},"field":"source","show_pie_chart":false,"query":"","show_data_table":true}},{"creator_user_id":"root","cache_time":10,"description":"Gesamt Meldungen","id":"7b23de7a-9730-4083-b176-69eba399a7c5","type":"SEARCH_RESULT_COUNT","config":{"timerange":{"type":"keyword","keyword":"midnight yesterday to midnight today"},"lower_is_better":true,"trend":true,"query":""}},{"creator_user_id":"root","cache_time":10,"description":"Meldungen Probleme","id":"de7e0cbc-bd93-402f-9586-ef1db016451e","type":"QUICKVALUES","config":{"timerange":{"type":"keyword","keyword":"midnight yesterday to midnight today"},"field":"source","show_pie_chart":false,"query":"(level:0 OR level:1 OR level:2 OR level:3 OR level:4)","show_data_table":true}}]}

And for one of the broken widgets:

{"id":"24aff977-bbc0-4d46-af00-2d67c7264a36","description":"Problem Meldungen","type":"SEARCH_RESULT_COUNT","config":{"timerange":{"type":"keyword","keyword":"midnight yesterday to midnight today"},"lower_is_better":true,"trend":true,"query":"(level:0 OR level:1 OR level:2 OR level:3 OR level:4)"},"cache_time":10,"creator_user_id":"root"}

Thank you,
Klaus.

@edmundoa

This comment has been minimized.

Member

edmundoa commented Jun 1, 2016

@tachtler Thank you for providing some examples! I will look into them and try to find that nasty bug.

In your particular case, what I saw after a quick look is that your dashboard is actually querying data from yesterday. Assuming that your data sources have the right time configuration, the data will not change during the day. Of course, I might be assuming too much.

Would you be so kind as to elaborate a bit more on when you experience this issue? I am specially interested if you see the data from yesterday changing during the day (you can verify by playing the "replay search" button in one of your widgets every now and then).

It would also be interesting to see if the dashboard data is updated when the day changes.

@edmundoa

This comment has been minimized.

Member

edmundoa commented Jun 1, 2016

@tachtler The only "issue" I could find with your dashboard is the one I mentioned before: the time range searches in messages from yesterday, and the data does not update unless you send messages with yesterdays' timestamp, or the day changes. Still, my dashboard was correctly asking for new data.

@tachtler

This comment has been minimized.

tachtler commented Jun 1, 2016

Hi Edmundo,

let me explain me, my dashboard a little bit more.

My intention is, when I came into my office in the morning, I will see how much messages at all and how much problem messages where generated the last 24 hours starting 2 days midnight before to 1 day midnight before, as well.

I know, that the data will not change during the actual day. BUT when I login to graylog, a few days, always the same message count will be shown. Only when I start the described procedure, the data will change and stay correct for the rest of the whole day.

At graylog before version 2.0.x every morning the message count was counted correct.

Can I sent you some more data, to help solving the problem?

Thank you,
Klaus.

@tachtler

This comment has been minimized.

tachtler commented Jun 1, 2016

Hi Edmundo,

my environment for the graylog server are:

  • OS: CentOS 7.2
  • Installed graylog-server (from el repo):
Name        : graylog-server
Arch        : noarch
Version     : 2.0.2
Release     : 1
Size        : 91 M
Repo        : installed
From repo   : graylog
Summary     : Graylog server
URL         : https://www.graylog.org/
License     : GPLv3
Description : Graylog server
Installed Packages
Name        : mongodb
Arch        : x86_64
Version     : 2.6.11
Release     : 1.el7
Size        : 131 M
Repo        : installed
From repo   : epel
Summary     : High-performance, schema-free document-oriented database
URL         : http://www.mongodb.org
License     : AGPLv3 and zlib and ASL 2.0
Installed Packages
Name        : elasticsearch
Arch        : noarch
Version     : 2.3.3
Release     : 1
Size        : 29 M
Repo        : installed
From repo   : elasticsearch-2.x
Summary     : Distribution: RPM
License     : (c) 2009

Greetings and 'Moin moin' to Hamburg...
Klaus.

@edmundoa

This comment has been minimized.

Member

edmundoa commented Jun 2, 2016

Thank you for providing some context, @tachtler! That's a perfect valid use case, although I'm not entirely sure it is the same situation as first described by @OlympiaLady.

Do you see any logs coming from Graylog server or Elasticsearch when this situation happen?

@tachtler

This comment has been minimized.

tachtler commented Jun 2, 2016

Hi Edmundo,

so, I can give you all log entrys from /var/log/graylog-server/server.log - generated when I

  1. login to graylog
  2. open the dashboard
  3. update the widget, so that the correct value will be displayed
----- Login to graylog ----
2016-06-02T15:27:54.187+02:00 INFO  [LdapConnector] LDAP group search base, id attribute or object class missing, not iterating over LDAP groups.
2016-06-02T15:27:54.203+02:00 ERROR [LdapUserAuthenticator] Error during LDAP user account sync. Cannot log in user <SECRET ;-)>
java.lang.NullPointerException
    at org.graylog2.users.UserImpl.setPermissions(UserImpl.java:176) ~[graylog.jar:?]
    at org.graylog2.users.UserServiceImpl.updateFromLdap(UserServiceImpl.java:232) ~[graylog.jar:?]
    at org.graylog2.users.UserServiceImpl.syncFromLdapEntry(UserServiceImpl.java:160) ~[graylog.jar:?]
    at org.graylog2.security.realm.LdapUserAuthenticator.doGetAuthenticationInfo(LdapUserAuthenticator.java:121) [graylog.jar:?]
    at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568) [graylog.jar:?]
    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:219) [graylog.jar:?]
    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:269) [graylog.jar:?]
    at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) [graylog.jar:?]
    at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) [graylog.jar:?]
    at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270) [graylog.jar:?]
    at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [graylog.jar:?]
    at org.graylog2.rest.resources.system.SessionsResource.newSession(SessionsResource.java:103) [graylog.jar:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_91]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_91]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_91]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_91]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
----- Open Dashbord -----
----- Unlock/Edit widget -----
----- Press Update-Button inside Edit widget -----
2016-06-02T15:28:35.876+02:00 INFO  [DashboardWidgetsResource] Updated widget <7b23de7a-9730-4083-b176-69eba399a7c5> on dashboard <57230775c7df8a3c2186d4cf>. Reason: REST request.
----- EOF -----

Next the LOG-Entrys from /var/log/elasticsearch/graylog.log

[2016-06-02 02:00:03,817][INFO ][cluster.metadata         ] [Lady Lotus] [graylog_26] creating index, cause [api], templates [graylog-internal], shards [1]/[0], mappings [message]
[2016-06-02 02:00:04,394][INFO ][cluster.routing.allocation] [Lady Lotus] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_26][0]] ...]).
[2016-06-02 02:00:04,795][INFO ][cluster.metadata         ] [Lady Lotus] [graylog_26] update_mapping [message]
[2016-06-02 02:00:15,778][INFO ][cluster.metadata         ] [Lady Lotus] [graylog_26] update_mapping [message]

Hope this can help?

When you need more information, tell me what to do...

Thank you
Klaus.

@edmundoa

This comment has been minimized.

Member

edmundoa commented Jun 3, 2016

@tachtler Unfortunately there's nothing that can help me figuring out what is causing the issue.

The last couple of days I have been paying special attention to some of our dashboards, and also trying to reproduce this, but I haven't seen any problems at all.

Let's try to get some responses from the server, and see if we can find out something with that. Here are the steps to do so. Don't hesitate asking if anything is not clear enough or you have some doubts during the process.

  1. Wait until the issue occurs, it's important that you don't update the widget at this point, we need to see what the server is responds
  2. Open a terminal in your computer
  3. Get the current value for one of the broken widgets. Using the IDs you already shared, this example would get the value of the widget "Problem Meldungen":
curl -XGET -v -H 'X-Requested-With: XMLHttpRequest' 'http://<username>:<password>@<graylog-server-host>:12900/dashboards/57230775c7df8a3c2186d4cf/widgets/24aff977-bbc0-4d46-af00-2d67c7264a36/value?pretty=true'

4 . Update the widget configuration from the web interface, as you usually workaround the issue
5. Repeat step 3, to get the "updated" value

Once the process is done, please share with us the results of doing steps 3 and 5.

Thank you!

@tachtler

This comment has been minimized.

tachtler commented Jun 3, 2016

Hi Edmundo,

STEP 3: Bevore updating:

* About to connect() to 127.0.0.1 port 12900 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 12900 (#0)
* Server auth using Basic with user 'geheim'
> GET /dashboards/57230775c7df8a3c2186d4cf/widgets/24aff977-bbc0-4d46-af00-2d67c7264a36/value?pretty=true HTTP/1.1
> Authorization: Basic cm9vdDpjYWxpbDFsYWt1JDA=
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:12900
> Accept: */*
> X-Requested-With: XMLHttpRequest
> 
< HTTP/1.1 200 OK
< X-Graylog-Node-ID: c3f06311-9fc4-40ad-9b78-452d7fd70191
< Vary: Accept-Encoding
< X-Runtime-Microseconds: 214105
< Content-Type: application/json
< Date: Fri, 03 Jun 2016 14:57:51 GMT
< Content-Length: 86
< 
{
  "result" : 1721,
  "took_ms" : 4,
  "calculated_at" : "2016-06-03T14:57:51.432Z"
* Connection #0 to host 127.0.0.1 left intact

When I click on the small "play" arrow inside the Dashboard view widget, i came to the graylog site and I can see the real search result. It will be for today 265!

STEP 4: update the widget with by unlock/edit the widget and pressing the update button
STEP 5: Repeat STEP3...
STEP 3 (once again): After update with the web application:

* About to connect() to 127.0.0.1 port 12900 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 12900 (#0)
* Server auth using Basic with user 'geheim'
> GET /dashboards/57230775c7df8a3c2186d4cf/widgets/24aff977-bbc0-4d46-af00-2d67c7264a36/value?pretty=true HTTP/1.1
> Authorization: Basic cm9vdDpjYWxpbDFsYWt1JDA=
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:12900
> Accept: */*
> X-Requested-With: XMLHttpRequest
> 
< HTTP/1.1 200 OK
< X-Graylog-Node-ID: c3f06311-9fc4-40ad-9b78-452d7fd70191
< Vary: Accept-Encoding
< X-Runtime-Microseconds: 162792
< Content-Type: application/json
< Date: Fri, 03 Jun 2016 15:05:29 GMT
< Content-Length: 86
< 
{
  "result" : 265,
  "took_ms" : 12,
  "calculated_at" : "2016-06-03T15:05:29.191Z"
* Connection #0 to host 127.0.0.1 left intact

The result from pressing the small "play" arrow and after the web application update, by Unlock/Edit the widget and press the update button are the same: 265!

Edmundo, all my widgets at this dashboard are calculating the results with the keyword midnight yesterday to midnight today, so where can I see, that my dashbord will do a update at general. Is it possible, that at my installation is anything wrong? Where can I check something like that?

Thank you so much, that you care about the/my problem!
Klaus.

@OlympiaLady

This comment has been minimized.

OlympiaLady commented Jun 3, 2016

Hi Edmundo,
I have the exact same behavior as tachler:

  1. The data is old when I open the dashboard.
  2. If I go into Unlock/Edit mode and change the "Cache time", let's say from 5 to 4 and click Update, The data refreshes and becomes current.
  3. The issue is only for one widget on my dashboard.
{
  "result" : {
    "total" : 24,
    "other" : 0,
    "terms" : {
      "dshswboly80850" : 24
    },
    "missing" : 0
  },
  "took_ms" : 8,
  "calculated_at" : "2016-06-03T16:14:51.967Z"
}

AFTER FORCED UPDATE BY CHANGING CACHE TIME:

{
  "result" : {
    "total" : 27,
    "other" : 0,
    "terms" : {
      "dshswboly80850" : 24,
      "dshswboly80854" : 3
    },
    "missing" : 0
  },
  "took_ms" : 180,
  "calculated_at" : "2016-06-03T16:18:14.310Z"
}

Sorry I have not been very useful, and yes, thank you Edmundo for your efforts. I will try to squeeze in more time to give you some more information.

Beth

@OlympiaLady

This comment has been minimized.

OlympiaLady commented Jun 3, 2016

widget_problem

@edmundoa

This comment has been minimized.

Member

edmundoa commented Jun 3, 2016

Thank you @tachtler and @OlympiaLady for your patience with this issue and all the good feedback provided!

I have found a problem with the way we handle caching of widgets using keyword searches that explains the issue you are seeing. When you update a widget, it's cache is deleted, and the time range updated next time the widget value is calculated, which explains why widgets work again after editing them.

As a temporary workaround, you may want to try using a relative time range, which should work fine. At least in your case, @OlympiaLady, it should be fairly easy to do: replay the search of your widget, and change the time range to use a relative one, and then select "Search in the last 2 hours" from the dropdown.

@edmundoa edmundoa added bug and removed cannot reproduce web labels Jun 3, 2016

@edmundoa edmundoa added this to the 2.0.3 milestone Jun 3, 2016

@OlympiaLady

This comment has been minimized.

OlympiaLady commented Jun 3, 2016

Here is my widget configuration:

{"id":"50b3898e-6d5e-48f7-97f7-330a48345490","description":"Web Activity Last 2 Hours","type":"QUICKVALUES","config":{"timerange":{"type":"keyword","keyword":"Last 2 hours"},"field":"source","query":"SourceName:IIS","show_data_table":true,"interval":"minute","show_pie_chart":true},"cache_time":6,"creator_user_id":"alberbm-a"}

I looked in the graylog log /var/log/graylog/server/current with
cat current | grep ERROR

2016-06-03_00:17:07.46484 2016-06-02 17:17:07,232 ERROR: org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy - Unknown index, cannot perform rotation
2016-06-03_00:17:10.88622 2016-06-02 17:17:07,467 ERROR: org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy - Cannot perform rotation at this moment.

I got several of these:
2016-06-03_05:01:54.07239 2016-06-02 22:01:52,366 ERROR: org.glassfish.jersey.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream.

2016-06-03_06:49:59.46724 2016-06-02 23:49:58,809 ERROR: org.graylog2.periodical.AlertScannerThread - Skipping alert check that threw an exception.

2016-06-03_06:50:26.58632 2016-06-02 23:50:26,585 ERROR: org.graylog2.periodical.IndexRetentionThread - Uncaught exception in periodical

2016-06-03_07:05:42.74369 2016-06-03 00:05:42,127 ERROR: org.glassfish.jersey.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream.
2016-06-03_09:34:13.83418 2016-06-03 02:34:13,402 ERROR:
org.graylog2.periodical.IndexRotationThread - Couldn't point deflector to a new index
2016-06-03_09:34:14.93025 2016-06-03 02:34:10,754 ERROR: org.graylog2.periodical.AlertScannerThread - Skipping alert check that threw an exception.

2016-06-03_09:47:06.75290 2016-06-03 02:47:06,312 ERROR: org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy - Unknown index, cannot perform rotation

2016-06-03_09:47:07.66930 2016-06-03 02:47:06,797 ERROR: org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy - Cannot perform rotation at this moment.

2016-06-03_11:08:59.13876 2016-06-03 04:08:58,676 ERROR: org.graylog2.periodical.AlertScannerThread - Skipping alert check that threw an exception.

2016-06-03_11:09:08.03709 2016-06-03 04:08:59,399 ERROR: org.graylog2.indexer.messages.Messages - Failed to index [303] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:

I got several of the below:

2016-06-03_16:21:09.24929 2016-06-03 09:21:08,461 ERROR:
org.glassfish.jersey.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream.

@OlympiaLady

This comment has been minimized.

OlympiaLady commented Jun 3, 2016

Thank you VERY much Edmundo. You are very appreciated!

Beth

@edmundoa

This comment has been minimized.

Member

edmundoa commented Jun 3, 2016

A small correction, the actual problem is not the caching itself, is that the keyword time range behaves differently as the relative time range, and subsequent calls to getFrom() and getTo() always return the same values. As those objects are kept in memory, and not reloaded from the database, the calculation ranges are always the same until the cache is invalidated.

@joschi joschi added S3 P2 labels Jun 6, 2016

edmundoa added a commit that referenced this issue Jun 6, 2016

Calculate keyword from and to values on the fly
Keyword time ranges may be in memory for a while, and the range they
represent may change over that time. Currently, they are also inconsistent
with the way relative ranges work, which are calculated when `getFrom` and
`getTo` are called.

These changes avoid those two problems by calculating the keyword from and
to values when the methods are called.

Fixes #2301

joschi added a commit that referenced this issue Jun 9, 2016

Calculate keyword from and to values on the fly (#2335)
Keyword time ranges may be in memory for a while, and the range they
represent may change over that time. Currently, they are also inconsistent
with the way relative ranges work, which are calculated when `getFrom` and
`getTo` are called.

These changes avoid those two problems by calculating the keyword from and
to values when the methods are called.

Fixes #2301

@joschi joschi closed this in 5f7b43c Jun 9, 2016

@OlympiaLady

This comment has been minimized.

OlympiaLady commented Jun 9, 2016

Thank you again to both of you!

@tachtler

This comment has been minimized.

tachtler commented Jun 10, 2016

Hi,

thank you for solving the problem, great so see, how the work is done in this project!

THANK YOU!
Klaus.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment