When calling multiple graphs, only 2 show up

[spacertechie]

Describe the bug
Saying "graf db:" used to show all available graphs. Now it displays all the titles but only shows the graphs for two.

To Reproduce
Steps to reproduce the behavior:

1. Create a dashboard named "db" with multiple graphs
2. Call hubot in Slack

Expected behavior
Usually, all graphs and their titles would be displayed.

Screenshots
https://imgur.com/a/bXoGuFM

Software:
Grafana: v6.3.X
Hubot: v3.3.2
Adapter: Slack

Additional context
This was working until the latest grafana update a week or two ago

[stephenyeargin]

Suspicion is that the other graphs aren't returning fast enough to be attached, which means we need to increase the TTL somehow to wait around for the rendering process to finish. Anything interesting in the bot's logs (HUBOT_LOG_LEVEL=debug is great for this, though verbose)?

[spacertechie]

ok thank you! will check out the logs

[spacertechie]

Ok! Got something promising:

[Mon Dec 16 2019 13:05:21 GMT-0500 (EST)] DEBUG Uploading file: 1722 bytes, content-type[text/html; charset=UTF-8]
[Mon Dec 16 2019 13:05:21 GMT-0500 (EST)] DEBUG SlackClient#send() room: , message: [object Object]
[Mon Dec 16 2019 13:05:23 GMT-0500 (EST)] DEBUG Uploading file: 1722 bytes, content-type[text/html; charset=UTF-8]
[Mon Dec 16 2019 13:05:24 GMT-0500 (EST)] DEBUG SlackClient#send() room: , message: [object Object]
[Mon Dec 16 2019 13:05:27 GMT-0500 (EST)] DEBUG Uploading file: 1722 bytes, content-type[text/html; charset=UTF-8]
[Mon Dec 16 2019 13:05:27 GMT-0500 (EST)] DEBUG SlackClient#send() room: , message: [object Object]
[Mon Dec 16 2019 13:05:29 GMT-0500 (EST)] DEBUG Uploading file: 1722 bytes, content-type[text/html; charset=UTF-8]
[Mon Dec 16 2019 13:05:29 GMT-0500 (EST)] DEBUG Uploading file: 1722 bytes, content-type[text/html; charset=UTF-8]
[Mon Dec 16 2019 13:05:29 GMT-0500 (EST)] DEBUG SlackClient#send() room: , message: [object Object]
[Mon Dec 16 2019 13:05:29 GMT-0500 (EST)] DEBUG SlackClient#send() room: , message: [object Object]
[Mon Dec 16 2019 13:05:34 GMT-0500 (EST)] DEBUG Uploading file: 1722 bytes, content-type[text/html; charset=UTF-8]
[Mon Dec 16 2019 13:05:34 GMT-0500 (EST)] DEBUG SlackClient#send() room: , message: [object Object]
[Mon Dec 16 2019 13:05:36 GMT-0500 (EST)] DEBUG Uploading file: 1722 bytes, content-type[text/html; charset=UTF-8]
[Mon Dec 16 2019 13:05:36 GMT-0500 (EST)] DEBUG Uploading file: 15664 bytes, content-type[image/png]
[Mon Dec 16 2019 13:05:36 GMT-0500 (EST)] DEBUG SlackClient#send() room: , message: [object Object]
[Mon Dec 16 2019 13:05:36 GMT-0500 (EST)] DEBUG Uploading file: 14997 bytes, content-type[image/png]

Those last 2 messages with 14k bytes are the successful images. It seems the failed images are being received as "text/html" and the images are "image/png"

[spacertechie]

Hi just wanted to check in? Are there any config changes we can try?

[stephenyeargin]

Unfortunately, I can't think of anything else. The contents of those HTML pages is likely a rendering error, perhaps from server memory running short? PhantomJS, the tool used in Grafana Server, is an inexact science for rendering those images -- it's a headless browser that attempts to run the provided JavaScript and return a PNG. If anything goes wrong in that process, it spits out an exit code and the endpoint returns a HTTP/500.

I'm guessing that going to the individual panels and rendering an image is working?

[spacertechie]

I see, thank you for checking it out. The server running hubot is healthy / overprovisioned, I think the bot just needs to sleep/wait after it requests images and before it tries to push images to Slack?

[spacertechie]

The issue was provisioning on the Grafana Server side! Thank you @stephenyeargin