Refresh overlay image when the dashboard refreshes manually or automatically

[falconhome]

Dear Support Team!

Thanks for supporting the very useful feature #84 contributed by @jreyesr! My request is a small new feature, based on my conversation with him.

Current implementation

The original code load the overlay image one time, when the dashboard load, and later on only the data points are refreshed.

Feature request

My request is to implement an "overlay configuration setting" where the dasboard developers can decide the overlay image is a static image, or periodically updated and has to be reload when the dasboard refreshed manually or automatically. Functionally, if this feature enabled on the dasboard, the system reload the image defined in the overlay url.

Background

In my case the background image preparation happen in a different application because of the complex math algorithm wich is not supported by Grafana. This is absolutelly understandable, because Grafana primary is a data visualization and exploration tool.

In the end of the line my app periodically rendering the overlay image and put it to the host on a same name, wich reachable over the overlay url. Now, if i want to load the most recent version for the image i have to reload the whole page or open a new tab which is extra load on a grafana server, and uncomfortable if i run the dasboard in kiosk mode.

And finally the reason why I proposed the config settings, because in many cases the users use static images and refreshing only the datapoints. In this case the periodically reload of the image is not required and i do not want to waste capacity for the renderer.

Thank you for your reply.

Br, Tibor.

[falconhome]

Some additional...
I added this code sniplets to

&the src/partials/editor.html behind the Overlay URL (from row 562)

• html sniplet - htmlsniplet.txt

&to the src/worldmap_ctrl.ts

• from row 63 - const def.
  enableReloadOverlay: false,

• if i understand well, i looking for the right place for some kind of this sniplet:
  if (ctrl.map.overlay && ctrl.panel.enableOverlay && ctrl.panel.enableReloadOverlay) { this.map.overlay.remove(); this.map.overlay = null; this.render(); ctrl.map.createOverlay(); }

The HTML tested - OK, but I cannot build the ts yet (im fighting with dockered grafana srv + yarn + js) sry.

@jreyeser: am i on a good way? Shall i ask you to help move forward?

[amotl]

Dear Tibor,

thank you very much for writing in and for proposing this feature.

Attaching to @jreyesr's comment at #84 (comment):

[Let's] get the official project maintainers involved (I am not a maintainer, by the way), and check if such an enhancement would be welcome or on scope for the project (I expect it would be, but it's probably a good idea to check with the maintainers first). You can mention that I am willing to implement the feature, since I developed the original overlay code, so there would be very little extra work for the maintainers.

We will be very happy to accept any kinds of improvements to this canonical fork of the original Grafana Worldmap Panel which supports the community of Grafana map lovers.

With kind regards,
Andreas.

[jreyesr]

@falconhome Yeah, those changes are pretty much everything that is required, apart from adding the new option to the README.

am i on a good way? Shall i ask you to help move forward?

Would you like to implement the feature yourself or are you OK with me doing it? I ask because "am i on a good way?" sounds like you want to do it yourself. I can also do it (in fact, I think I have a working implementation).

[falconhome]

@jreyesr Ups. my English grammar need some refreshment.. :) i wanted to say i put this things together for you (if you can use it). So if you have a time to put this changes into your code and make a test build for me i will be very thankful.

[jreyesr]

Yes, I can use the code above. I have submitted a PR on #99. Take a look below for a test: the image is swapped every five seconds, and clicking the Refresh button updates it. It works the same with the automatic refresh, since the two refresh methods share functionality.

[falconhome]

@jreyesr: This is the function what i requested! Thank you so much!

[falconhome]

Thank you all of your work!
2 days and this feature request is on the way to use the community!
Five star!

[jreyesr]

@falconhome You may wish to wait for a bit before closing this issue. The functionality is not included yet, since #99 is not merged. After the PR is merged, this issue will be closed automatically, since I included the magic words "closes #97" in the PR description.
Later, the maintainers must create a new release that includes the functionality (see #84 (comment), for example). Once the release is created, you will be able to install the plugin normally (through grafana-cli plugins install or by manually downloading the folder)

[falconhome]

@jreyesr: Thank you! I reopen it (sorry for this)
could you send me a priv message? i do not want to flame this topic ;)

[falconhome]

@jreyesr, @amotl
I downloaded the latest plugin source (ZIP) & try to build this originals with yarn, but i got Linting errors.
Is it possible i use wrong source? (the direct git pull has a same error without any changes)

Linting errors.txt

Thank you so much.
Tibor

[amotl]

Dear Tibor,

I will have to look into the linting errors, thanks for the heads up! I will also try to issue a new release soon. Thanks a stack for implementing this feature so quickly, @jreyesr!

With kind regards,
Andreas.

[jreyesr]

@amotl You're welcome!

[falconhome]

@jreyesr: Hi! Do you have any idea why i got this linting error? (i tired to compile other grafana plugin source and that finished correctly) or if you compiled it could you send me a zip or a link where i can download the builded version? My email address in my profile.
Thanks a lot!
Tibor

[jreyesr]

@falconhome I have opened an issue to track the linting error on #101

[falconhome]

@jreyesr: thanks for the changes, the build now can finish with no errors! Halleluja! :)
I installed the "new plugin" under grafana, the new option is arrive under overlay in edit mode, perfect, but the image does not refreshing.
I tested under edge, chrome & firefox all the same.
This could be some kind of cacheing issue?
Any idea what i can do?
Thanks

[jreyesr]

@falconhome Yeah, I suspect it is a caching issue, as it also happened to me. Here's a quick check:

1. Open the developer tools (F12 on Chrome) on the Grafana tab, go to the Network tab, refresh the page (with F5) and look for the image in the list of requests. You can filter the requests to only show images by clicking the "Img" option near the top, next to the Filter input field.
2. Click the Refresh Dashboard button (not the browser Refresh!) or wait for the autorefresh period, and see if another image request appears. If it doesn't (or if it appears but the Size column says "(disk cache)" or "(memory cache)"), the browser is caching the image and not making a new request.
3. You can also check the access logs on the server (there should be a way to enable logging each received request). The logs should show a request to the image URL every time the dashboard refreshes. If there are no new requests, it means that the browser is not sending the requests, and it is most likely a caching issue.

Here's a capture of the Network tab, when using a server that sets caching headers (I used Python3's python3 -m http.server, but I expect most servers to work similarly by default, since they will probably attempt to cache things, to reduce load):

Notice that, when reloading, the only new requests are query?db=..., which fetch the data, but there are no new image fetches. When running the script on https://stackoverflow.com/a/28708920 (as ./serveit.py), these are the requests:

Now, on each reload, there is a request to localhost:8000/img along each request to query?db=.... The modified server adds the following response headers, which force the browser to make a fresh request every time:

Sadly, that issue is outside the power of Grafana and grafana-map-panel, I think. That must be fixed in the server, by setting the proper cache headers: either disabling caching completely, or somehow setting the expiration date to be the time at which you know that a new image will be available (for example, if the images are generated at 12:00 every night, set the Expires header to next midnight, which will signal to the browser that the current image will go stale at midnight).

[falconhome]

Hi @jreyesr!

I configured my web server cache control.
If i directly download the image, i can see the cache control works properly, the max-age came with 0. (perfect)

But, if i check the same image over grafana map panel, it looks the cache-control setting does not exist in the "Request header".

Shomewhere lost in space... :( Now my hands up in the air...

I do not think this is outside of the grafana scope, but now i cannot know which way will be better...

[amotl]

Dear Tibor,

thank you for reporting your observations. I am slightly confused: Wouldn't you have to configure your webserver's response headers accordingly as outlined by @jreyesr? Currently, I don't see anything like that happening when looking at both screenshots you shared with us.

With kind regards,
Andreas.

[jreyesr]

@falconhome As @amotl said, I am also quite sure that the response would need to have the cache headers. In your first screenshot, the Cache-Control header is set on the request, not on the response. You seem to be using Apache, so something like Header set Cache-Control "max-age=0, no-cache, no-store" should work, as long as you apply it on the proper files (maybe through FilesMatch?).

The headers should appear under the "Response Headers" list both when you access the image directly (by writing its URL on the address bar) and when inspecting the requests fired off by Grafana.

This is my Developer Tools when accessing the image directly:

[falconhome]

Dear Andreas!

I needs to be more detailed, sorry!
I have Apache web server - Server version: Apache/2.4.46 (Unix), Server built: Apr 10 2021 12:28:18 -,
where i done @jreyesr proposed settings in .htaccess. My configuration, based on Knowledge Base of A2Hosting - https://www.a2hosting.com/kb/developer-corner/apache-web-server/turning-off-caching-using-htaccess. (i copy pasted it)

Appache restart many time done.

And so the very interesting thing:
If i get directly the "overlay" image from my apache web server, the "Cache-Control: max-age=0" visible in a Request Header, see below picture, this means the cache control changes propagated by the apache! (you can se, the chrome "disable cache" not playing - i switched off, the link is pointing directly to the image file...)

BUT, the interesting is, if i check inside the Grafana map-panel the overlay image Request header has changed, and the Cache-Control: max-age=0 disappear.

So here i am, and i do not understand, sorry.
I missed something, shoewhere?

Thanks
Tibor

[falconhome]

**Andreas, @jreyesr **
I found the problem, details in a next comment flew minutes!

[falconhome]

So, if somebody use pytohon3 http.server for hosting the "overlay image" please follow @jreyesr configuration description!

For Apache web servers please follow the next steps, for the proper working of the automatic overlay refresh:
(The tested server: Server version: Apache/2.4.46 (Unix), Server built: Apr 10 2021 12:28:18 )

Step1: Check your httpd.conf file!
Enable the following modules (remove #) :

LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so

(!) both required

Step2: Add the following code to your httpd.conf file, or to the .htaccess based on your http management model
httpd.conf extension.txt
(Source: https://www.a2hosting.com/kb/developer-corner/apache-web-server/turning-off-caching-using-htaccess)

Step3: Restart your webserver (systemctl restart httpd)

Evaluation of the configuration changes
Load your overlay image directly from the webserver like: http://yourservername/yourimagedirectory/youroverlayimage.jpg

For Chrome users:

Left Click on the image and select Inspect
Select Network panel & refresh your image CTRL + R
Your image name will be visible in the right side of the Inspect area - click to it

If all changes in your webserver works, you have to see the followings under the Response Headers

Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache

If all done, the image will be refreshing properly. :)