Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Grafana data proxy should drop origin header #13328

Closed
ssozonoff opened this issue Sep 18, 2018 · 12 comments

Comments

Projects
None yet
5 participants
@ssozonoff
Copy link

commented Sep 18, 2018

Using Elasticsearch 5.6 the datasource (in proxy mode) passes tests and connects properly however its impossible to perform any query in a panel. Seems like Grafana proxying is POST'ing requests to ES instead of performing a GET which is causing a 502.

Grafana 5.2.2

t=2018-09-18T20:15:03+0000 lvl=info msg="Proxying incoming request" logger=data-proxy-log userid=1 orgid=1 username=admin datasource=elasticsearch uri=/api/datasources/proxy/2/_msearch method=POST body="{\"search_type\":\"query_then_fetch\",\"ignore_unavailable\":true,\"index\":\"my_stats_2018\",\"max_concurrent_shard_requests\":256}\n{\"size\":500,\"query\":{\"bool\":{\"filter\":[{\"range\":{\"@timestamp\":{\"gte\":\"1521407703352\",\"lte\":\"1537301703352\",\"format\":\"epoch_millis\"}}},{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"business.objId:6435\"}}]}},\"sort\":{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}},\"script_fields\":{},\"docvalue_fields\":[\"@timestamp\"]}\n"

@MrSantamaria

This comment has been minimized.

Copy link

commented Sep 18, 2018

Currently using ES 6.3, my queries are also being sent as POST and they are working. It seems like the issue is with the proxy setup.

@ssozonoff

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

You are right POST is permitted to ES. So I dont understand what the problem is then. Why the Datasource validates on passes the self test yet all queries fail with a 502

@msteggink

This comment has been minimized.

Copy link

commented Sep 19, 2018

Did it work before? What type of queries are you firing? Any hints in the ElasticSearch logging? A 502 might indicate faulty queries. Are you trying to do queries on text? You might need to use the keyword:
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/fielddata.html

@ssozonoff

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

Has never worked. Performing the same query from Curl works fine using POST method so I dont understand.

@torkelo

This comment has been minimized.

Copy link
Member

commented Sep 19, 2018

Do you have any error returned by Elasticsearch? Open Query inspector to view request and response from elasticsearch

@ssozonoff

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

Getting a 403 but dont understand why. Have a Wireshark trace and cant spot the issue yet!! As mentioned same query works in Curl. I am clearly missing something.

One option is something in the request headers that ES is not happy about.

[wireshark]https://imagizer.imageshack.com/v2/800x600q90/921/gYyrpJ.jpg

@marefr

This comment has been minimized.

Copy link
Member

commented Sep 19, 2018

Please enable ES debugging in Grafana and check Grafana server log:

[log]
filters = tsdb.elasticsearch:debug,tsdb.elasticsearch.client:debug
@ssozonoff

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

Following on from my previous comment this could well be a CORS thing.

@ssozonoff

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

My fix for this has been to strip the Origin header in nginx.

@marefr

This comment has been minimized.

Copy link
Member

commented Sep 20, 2018

@ssozonoff glad you solve it. So the problem was that you nginx proxy added the origin header?

@marefr marefr closed this Sep 20, 2018

@ssozonoff

This comment has been minimized.

Copy link
Author

commented Sep 20, 2018

No the problem is the browser adds the Origin and the DS Proxy simply passes it along, which according to Grafana documentation is not what is supposed to happen in Server Mode. Documentation clearly states that using "Server Mode" avoid CORS issues... which it would if the DS Proxy would not pass on the Origin header it receives from the brower.

i.e. doing this same call in Curl works fine.

So my hack is to remove the header using Nginx

@marefr marefr reopened this Sep 20, 2018

@marefr

This comment has been minimized.

Copy link
Member

commented Sep 20, 2018

Just verified and seems Chrome include origin header even though it's not a CORS request where Firefox don't. Agree with you that the origin header should be removed in Grafana proxy.

@marefr marefr added this to the 5.4 milestone Sep 20, 2018

@marefr marefr changed the title Elasticsearch Datasource OK but queries in panel fail - 502 [Bug] Grafana data proxy should drop origin header Sep 20, 2018

roidelapluie added a commit to roidelapluie/grafana that referenced this issue Nov 8, 2018

Remove Origin and Referer headers while proxying requests
Fix grafana#13949
Fix grafana#13328

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

roidelapluie added a commit to roidelapluie/grafana that referenced this issue Nov 8, 2018

Remove Origin and Referer headers while proxying requests
Fix grafana#13949
Fix grafana#13328

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

@torkelo torkelo modified the milestones: 5.4, 5.3.4 Nov 11, 2018

bergquist added a commit to bergquist/grafana that referenced this issue Nov 12, 2018

Remove Origin and Referer headers while proxying requests
Fix grafana#13949
Fix grafana#13328

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
(cherry picked from commit 62417ca)

marefr added a commit that referenced this issue Nov 13, 2018

@marefr marefr removed the area/datasource label Mar 30, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.