Graylog Enterprise Report generation failing

[ahmedsajid]

First of all, I appreciate all the work the Graylog Community and maintainers put into it to make it a GREAT product.

In my current setup, Graylog Cluster of 3 nodes is setup as follows:

Client -> Load balancer (443) -> Nginx (https 9000) -> Graylog (https 9001)

Client browser accesses Graylog via IP on Load Balancer as url https://graylog.example.com/ URL. This is also the http_external_uri. graylog.example.com is not a DNS entry that server can resolve but it works on the client's machines.

Then there's http_publish_uri set to https://<NodeIP>:9001/.

All of the functionality works as expected. However after installation of the Graylog Small Business Enterprise License for the reporting feature, whenever I try to generate report through the UI, I get following error:

2021-03-15T17:05:29.672-04:00 ERROR [ReportRenderJob] Error during report generation: com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 3 attempts. - browser logs: 
2021-03-15T17:05:29.672-04:00 ERROR [ReportRenderJob] Failed to generate report: 
com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 3 attempts.
        at com.github.rholder.retry.Retryer.call(Retryer.java:174) ~[graylog.jar:?]
        at org.graylog.plugins.report.scheduler.ReportRenderJob.doRun(ReportRenderJob.java:74) [graylog-plugin-enterprise-3.3.9.jar:?]
        at org.graylog.plugins.report.scheduler.ReportRenderJob.run(ReportRenderJob.java:60) [graylog-plugin-enterprise-3.3.9.jar:?]
        at org.graylog.plugins.report.scheduler.ReportRenderSystemJob.execute(ReportRenderSystemJob.java:25) [graylog-plugin-enterprise-3.3.9.jar:?]
        at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:241) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_282]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.graylog.plugins.report.rest.chromedriver.ChromeDriverException: Render engine is unable to fully generate requested report, aborting. Browser logs: 
        [BROWSER]: [2021-03-15T17:02:29-0400] [SEVERE] https://graylog.example.com/assets/vendor.8d6aa5835c8302c41e2f.js - Failed to load resource: net::ERR_NAME_NOT_RESOLVED
        [BROWSER]: [2021-03-15T17:02:29-0400] [SEVERE] https://graylog.example.com/assets/polyfill.7b2d519528c4e87cbc34.js - Failed to load resource: net::ERR_NAME_NOT_RESOLVED
        [BROWSER]: [2021-03-15T17:02:29-0400] [SEVERE] https://graylog.example.com/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.0f2e4cf8cefdf620a71d.js - Failed to load resource: net::ERR_NAME_NOT_RESOLVED
        [BROWSER]: [2021-03-15T17:02:29-0400] [SEVERE] https://graylog.example.com/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.2148ca040de9a7a73c28.js - Failed to load resource: net::ERR_NAME_NOT_RESOLVED
        [BROWSER]: [2021-03-15T17:02:29-0400] [SEVERE] https://graylog.example.com/assets/builtins.7b2d519528c4e87cbc34.js - Failed to load resource: net::ERR_NAME_NOT_RESOLVED
        [BROWSER]: [2021-03-15T17:02:29-0400] [SEVERE] https://graylog.example.com/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.2d424d9fac3bf38109fc.js - Failed to load resource: net::ERR_NAME_NOT_RESOLVED
        [BROWSER]: [2021-03-15T17:02:29-0400] [SEVERE] https://graylog.example.com/assets/app.7b2d519528c4e87cbc34.js - Failed to load resource: net::ERR_NAME_NOT_RESOLVED
        [BROWSER]: [2021-03-15T17:02:29-0400] [SEVERE] https://graylog.example.com/config.js - Failed to load resource: net::ERR_NAME_NOT_RESOLVED
        [BROWSER]: [2021-03-15T17:02:29-0400] [SEVERE] https://graylog.example.com/assets/plugin/org.graylog.aws.AWSPlugin/plugin.org.graylog.aws.AWSPlugin.0898b5f648fe40b44913.js - Failed to load resource: net::ERR_NAME_NOT_RESOLVED
        at org.graylog.plugins.report.api.ReportService.fetchPdf(ReportService.java:260) ~[?:?]
        at org.graylog.plugins.report.scheduler.ReportRenderJob.lambda$doRun$0(ReportRenderJob.java:64) ~[?:?]
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[graylog.jar:?]
        at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[graylog.jar:?]
        ... 12 more

According to the doc as I understood I need to adjust the report_render_uri which I have set to https://<NodeIP>:9001/.

Now I can solve this problem by adding entry for graylog.example.com in /etc/hosts file but and I also want to know if report_render_uri config if its working as expected or not? Did I misconfigure anything?

Here's Nginx configuration for Graylog. Not sure if that makes a difference.

server {
    listen 9000 ssl;

    server_name graylog.example.com;



    root /usr/share/nginx/html;

    index index.html index.htm;

    access_log /var/log/nginx/graylog_example_com-access.log;
    error_log /var/log/nginx/graylog_example_com-error.log error;


    ssl_certificate_key /etc/nginx/tls/key.pem;
    ssl_certificate /etc/nginx/tls/cert_chain.pem;

    location / {
      proxy_read_timeout  900;
      proxy_set_header    Host $http_host;
      proxy_set_header    X-Forwarded-Host $host;
      proxy_set_header    X-Forwarded-Server $host;
      proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_pass          https://127.0.0.1:9001;
    }

}

Expected Behavior

Report plugin to use report_render_uri for report generation.

Current Behavior

Report plugin seems to use http_external_uri for report generation.

Possible Solution

Steps to Reproduce (for bugs)

1. Install Graylog Open source.
2. Obtain Small Business License
3. Install plugin package
4. Setup reporting following the doc https://docs.graylog.org/en/3.3/pages/reporting/setup.html
5. Setup a Test report.
6. Click on Send Report Now to trigger generation of the particular report to be emailed after completion.

Context

Trying to configure scheduled reports to be delivered via email.

Your Environment

• Graylog Version: 3.3.9
• Java Version: java-1.8.0-openjdk-headless-1.8.0.282.b08-1.el7_9.x86_64
• Elasticsearch Version: 5.6.16
• MongoDB Version: 3.6.21
• Operating System: RHEL 7.9
• Browser version: Firefox 86 on Ubuntu Linux 20.04.2

[dennisoelkers]

Hey @ahmedsajid,

thanks for reporting this! Your issue was well-written, contained the right details and helped a lot debugging this problem.

While report_render_uri was meant to solve your use case, it seems like we have an issue with in-html references to assets of the web if, which result in using the value of the http_publish_uri instead.

For now, a workaround would be to make graylog.example.com resolvable and accessible from the Graylog server itself. We will work on this issue and fix it one of the next releases, so you can get rid of the workaround after that.

[ahmedsajid]

Hi @dennisoelkers,

You are very welcome. I understand the pain of trying to troubleshoot an issue without the other party providing the full detail. And I'm conscious of community effort here and I didn't want to create additional work without doing due diligence and thinking its a problem with the software while it might be a user error.

I appreciate your rapid response on this.

I'll add the DNS entry as a workaround for now.

Thank you!

[ahmedsajid]

If it helps others with a weird setup such as mine, the following workaround did the job for me.
I had to change report_render_uri to https://graylog.example.com to make it work and pointing to the LB IP in /etc/hosts file.
Otherwise I kept getting CORS errors when trying to point to local node IP with port in /etc/hosts file.

[dennisoelkers]

@ahmedsajid: Starting with 4.1 (currently in beta), you should be able to remove the report_render_uri and http_external_uri configuration settings. We are now generating relative URLs for the web interface assets as well as the backend API. (See #10726 for details)

[euidzero]

I wonder if this bug is not somehow back.

I'm using 4.2.1 and my config looks like :
http_external_uri = https://mydomain.com/graylog
http_publish_uri = http://10.0.0.2:9000/

10.0.0.2:9000 -> http and IP only reverse-proxy -> CDN (resolves for mydomain.com and does the SSL)

When generating report the chrome_debug.log shows that the resources are fetched from

http://10.0.0.2:9000/graylog/ instead of http://10.0.0.2:9000/.

Of course this fails as the server sends the http graylog "homepage" instead of the actual resource:

expected behavior :
the chrome report generation plugin should fetch : http://10.0.0.2:9000/config.js

actual behavior :
it fetches http://10.0.0.2:9000/graylog/config.js and reveive the content of the homepage (http://10.0.0.2:9000/)

Same if I do not set http_publish_uri.