[Actions] email action fails due to enabled HTTP proxy in kibana #92949

jagjag · 2021-02-26T02:39:11Z

Kibana version:
7.10.2
Elasticsearch version:
7.10.2
Server OS version:
Redhat Linux 7.9
Browser version:
Chrome 88.0.4324.190
Browser OS version:
Windows 10
Original install method (e.g. download page, yum, from source, etc.):
yum
Describe the bug:
Alerting mail action goes failed due to enabled HTTP proxy in kibana

Steps to reproduce:

add the proxy parameter to kibana.yml
add mail action and webhook action in Alerts Connectors
add Connectors to some alert item

Expected behavior:
both mail and webhook (with proxy config )alert action works

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Feb 26 06:26:36 gctn-elk-p02 kibana[14790]: {"type":"log","@timestamp":"2021-02-25T22:26:36Z","tags":["error","plugins","taskManager","taskManager"],"pid":14790,"message":"Task actions:.email \"847b6820-77b8-11eb-8ad4-4bcc303d04e7\" failed: Error: 发送电子邮件时出错"}  
#  发送电子邮件时出错  =  Error sending email

kibana config:

server.port: 80
server.host: "0.0.0.0"
elasticsearch.hosts: ["https://ip:9202"]
elasticsearch.username: "kibana"
elasticsearch.password: "xxxxx"
xpack.encryptedSavedObjects.encryptionKey : "something_at_least_32_characters"
xpack.security.encryptionKey: "something_at_least_32_characters"
xpack.actions.proxyUrl: "http://some-proxy-ip:8080"
xpack.actions.proxyRejectUnauthorizedCertificates: false
xpack.actions.rejectUnauthorized: false
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/elasticsearch-ca.pem"]
elasticsearch.ssl.verificationMode : certificate
i18n.locale: zh-CN

Any additional context:

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-02-26T16:07:31Z

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

pmuellr · 2021-03-01T16:05:55Z

Could you try execution the connector from it's "Test" tab? You may see more diagnostic information.

Also curious about your email settings - server, port, secure setting

Also, I'd have to look up the exact details on use of the proxy for email, but I believe your proxy will have to support connecting to an SMTP server via CONNECT, and it's not clear to me that every proxy will do this.

Lastly, there is a bug in Kibana < 7.11 where if the proxy protocol and target protocol are not the same (http/https), then the connection will not succeed as we try to connect to the target with the proxy's protocol. In this case, with an http proxy, we may be running into that error.

pmuellr · 2021-03-02T14:07:03Z

Another thing I'm wondering is if the intention is to NOT use the http proxy with the email connector. I was frankly a little surprised that the node_mailer package we use for mailing supported an http proxy. But I'm thinking that folks using an http proxy may also be expecting smtp to NOT use a proxy, because their proxy doesn't support ~~HTTP~~ SMTP.

If so, we'll need some separate setting for the email connector to ignore/respect the proxy setting.

gmmorris · 2021-03-02T15:41:59Z

Removing the Bug label unti lwe can confirm it isn't a configuration problem

jagjag · 2021-03-04T01:00:53Z

@pmuellr

hi, thank you for your reply. I reproduced the issue and got the same error last night. please check the log.
and I also tried slack please check the screenshot below. thank you.

kibana log

Mar 04 00:05:51 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T16:05:51Z","tags":["warning","plugins","actions","actions"],"pid":191768,"message":"action execution failure: .email:38a8d08a-7c03-4dda-834a-08a62d22ccf6: gcmail: error sending email: Invalid response from proxy: 403"}
Mar 04 00:05:51 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T16:05:51Z","tags":["error","plugins","taskManager","taskManager"],"pid":191768,"message":"Task actions:.email \"52f18740-7c3a-11eb-ad55-435dadb3735f\" failed: Error: error sending email"}
Mar 04 00:36:36 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T16:36:36Z","tags":["warning","plugins","actions","actions"],"pid":191768,"message":"action execution failure: .email:38a8d08a-7c03-4dda-834a-08a62d22ccf6: gcmail: error sending email: Invalid response from proxy: 403"}
Mar 04 00:36:36 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T16:36:36Z","tags":["error","plugins","taskManager","taskManager"],"pid":191768,"message":"Task actions:.email \"9ee61ef0-7c3e-11eb-ad55-435dadb3735f\" failed: Error: error sending email"}
Mar 04 01:22:46 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T17:22:46Z","tags":["warning","plugins","actions","actions"],"pid":191768,"message":"action execution failure: .email:38a8d08a-7c03-4dda-834a-08a62d22ccf6: gcmail: error sending email: Invalid response from proxy: 403"}
Mar 04 01:22:46 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T17:22:46Z","tags":["error","plugins","taskManager","taskManager"],"pid":191768,"message":"Task actions:.email \"107dce40-7c45-11eb-b00a-b59568b77a18\" failed: Error: error sending email"}
Mar 04 01:53:31 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T17:53:31Z","tags":["warning","plugins","actions","actions"],"pid":191768,"message":"action execution failure: .email:38a8d08a-7c03-4dda-834a-08a62d22ccf6: gcmail: error sending email: Invalid response from proxy: 403"}
Mar 04 01:53:31 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T17:53:31Z","tags":["error","plugins","taskManager","taskManager"],"pid":191768,"message":"Task actions:.email \"5c752510-7c49-11eb-b00a-b59568b77a18\" failed: Error: error sending email"}
Mar 04 02:08:52 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T18:08:52Z","tags":["warning","plugins","actions","actions"],"pid":191768,"message":"action execution failure: .email:38a8d08a-7c03-4dda-834a-08a62d22ccf6: gcmail: error sending email: Invalid response from proxy: 403"}
Mar 04 02:08:52 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T18:08:52Z","tags":["error","plugins","taskManager","taskManager"],"pid":191768,"message":"Task actions:.email \"82551ea0-7c4b-11eb-ad55-435dadb3735f\" failed: Error: error sending email"}
Mar 04 02:24:16 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T18:24:16Z","tags":["warning","plugins","actions","actions"],"pid":191768,"message":"action execution failure: .email:38a8d08a-7c03-4dda-834a-08a62d22ccf6: gcmail: error sending email: Invalid response from proxy: 403"}
Mar 04 02:24:16 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T18:24:16Z","tags":["error","plugins","taskManager","taskManager"],"pid":191768,"message":"Task actions:.email \"a84d0d00-7c4d-11eb-b00a-b59568b77a18\" failed: Error: error sending email"}
Mar 04 04:27:17 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T20:27:17Z","tags":["warning","plugins","actions","actions"],"pid":191768,"message":"action execution failure: .email:38a8d08a-7c03-4dda-834a-08a62d22ccf6: gcmail: error sending email: Invalid response from proxy: 403"}
Mar 04 04:27:17 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-03T20:27:17Z","tags":["error","plugins","taskManager","taskManager"],"pid":191768,"message":"Task actions:.email \"d795f070-7c5e-11eb-b00a-b59568b77a18\" failed: Error: error sending email"}

slack

Slack a message by curl with proxy config is ok on the kibana server

and got an error on the kibana connector test.

slack error message:

Mar 04 08:47:39 gcdc-elastic01 kibana[191768]: {"type":"log","@timestamp":"2021-03-04T00:47:39Z","tags":["warning","plugins","actions","actions"],"pid":191768,"message":"action execution failure: .slack:7596f858-69cd-46e0-835f-494ab8f419ae: GCLslack: error posting a slack message, retry later; retry: true"}
Mar 04 08:47:40 gcdc-elastic01 kibana[191768]: {"type":"response","@timestamp":"2021-03-04T00:47:39Z","tags":[],"pid":191768,"method":"post","statusCode":200,"req":{"url":"/api/actions/action/7596f858-69cd-46e0-835f-494ab8f419ae/_execute","method":"post","headers":{"content-length":"121","kbn-version":"7.10.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://IP:5601","referer":"http://IP:5601/app/management/insightsAndAlerting/triggersActions/connectors","accept-encoding":"gzip, deflate","accept-language":"zh-CN,zh;q=0.9","via":"1.1 ZY-WSG 0A003C4F","host":"ip:5601"},"remoteAddress":"ip","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36","referer":"http://IP:5601/app/management/insightsAndAlerting/triggersActions/connectors"},"res":{"statusCode":200,"responseTime":359,"contentLength":9},"message":"POST /api/actions/action/7596f858-69cd-46e0-835f-494ab8f419ae/_execute 200 359ms - 9.0B"}

pmuellr · 2021-03-04T13:34:11Z

There's a bug in versions of Kibana < 7.11 where if the proxy protocol and target protocol don't match, the resulting HTTP request is unlikely to work, because the call will be made with the protocol of the proxy and not the target.

That would happen in your case above with Slack, if xpack.actions.proxyUrl is still set to "http://some-proxy-ip:8080", since slack uses an https protocol and the proxy uses an http protocol.

That bug is fixed in 7.11.0.

pmuellr · 2021-03-04T13:37:22Z

The original question concerned the email action and proxy. Is the customer's proxy set up to handle SMTP connections? Not clear to me that every http/s proxy can handle those, and we're wondering if we need to support a mode where:

Kibana is configured with xpack.actions.proxyUrl
that proxy doesn't support handling SMTP connections
customer wants to use the email action

In that case, I don't think the email action will work, so we'll need to do >something< ... more config, or change the email action to allow ignoring the proxy setting, etc.

simioa · 2021-03-09T14:11:39Z

We have the same Issue when we set the proxy.
In our case the Proxy is configured to only allow access to external services and only after explicitly granting access to them.
When we now try to access internal Services, for example the internal mail server, the connector times out.
Same happens with internal Webhooks, etc. when the Proxy is defined.

Maybe having more granular Configuration Options or the possibility to define the Proxy per Connector would fix this problem.

pmuellr · 2021-03-09T15:10:18Z

@simioa I just added a comment to our issue on per-connector network customizations here - #80120 (comment)

Feel free to add more detail / let me know if something I wrote isn't correct.

pmuellr · 2021-03-24T16:29:25Z

For the per-connector network customizations, it feels like the proxy info is not a good fit here, and doesn't align with our current kibana config dealing with proxies: xpack.actions.proxyUrl, xpack.actions.proxyHeaders, and xpack.actions.proxyRejectUnauthorizedCertificates. We also have a config xpack.actions.allowedHosts which isn't proxy specific, but does deal in hostnames, and presumably some config for proxy bypass would be based on hostnames, so we use the same patterns for the config, as appropriate

Given all that, I think we should add two new configs:

xpack.actions.proxyBypassHosts - list of hostnames that should NOT use the proxy, others will use the proxy
xpack.actions.proxyOnlyHosts - list of hostnames that should ONLY use the proxy, others will not use the proxy

The config keys are mutually exclusive - only one can be set.

My first thought on cloud support is that we likely won't need it, because it doesn't seem like a cloud-deployed Kibana could connect to a customer-provided proxy. But with the newish private networking support, seems possible, or perhaps possible in the near future, so ... I think it should be cloud enabled.

resolves elastic#92949 This PR adds two new Kibana config keys to further customize when the proxy is used when making HTTP requests. Prior to this PR, if a proxy was set via the `xpack.actions.proxyUrl` config key, all requests would be proxied. Now, there's a further refinement in that hostnames can be added to the `xpack.actions.proxyBypassHosts` and `xpack.actions.proxyBypassHosts` config keys. Only one of these config keys can be used at a time. If the target URL hostname of the HTTP request is listed in the `proxyBypassHosts` list, the proxy won't be used. If the target URL hostname of the HTTP request is **NOT** listed in the `proxyOnlyHosts` list, the proxy won't be used. Depending on the customer's environment, it may be easier to list the hosts to bypass, or easier to list the hosts that should only be proxied, so they can choose either method.

resolves elastic#92949 This PR adds two new Kibana config keys to further customize when the proxy is used when making HTTP requests. Prior to this PR, if a proxy was set via the `xpack.actions.proxyUrl` config key, all requests would be proxied. Now, there's a further refinement in that hostnames can be added to the `xpack.actions.proxyBypassHosts` and `xpack.actions.proxyOnlyHosts` config keys. Only one of these config keys can be used at a time. If the target URL hostname of the HTTP request is listed in the `proxyBypassHosts` list, the proxy won't be used. If the target URL hostname of the HTTP request is **NOT** listed in the `proxyOnlyHosts` list, the proxy won't be used. Depending on the customer's environment, it may be easier to list the hosts to bypass, or easier to list the hosts that should only be proxied, so they can choose either method.

…#95365) resolves #92949 This PR adds two new Kibana config keys to further customize when the proxy is used when making HTTP requests. Prior to this PR, if a proxy was set via the `xpack.actions.proxyUrl` config key, all requests would be proxied. Now, there's a further refinement in that hostnames can be added to the `xpack.actions.proxyBypassHosts` and `xpack.actions.proxyOnlyHosts` config keys. Only one of these config keys can be used at a time. If the target URL hostname of the HTTP request is listed in the `proxyBypassHosts` list, the proxy won't be used. If the target URL hostname of the HTTP request is **NOT** listed in the `proxyOnlyHosts` list, the proxy won't be used. Depending on the customer's environment, it may be easier to list the hosts to bypass, or easier to list the hosts that should only be proxied, so they can choose either method.

…elastic#95365) resolves elastic#92949 This PR adds two new Kibana config keys to further customize when the proxy is used when making HTTP requests. Prior to this PR, if a proxy was set via the `xpack.actions.proxyUrl` config key, all requests would be proxied. Now, there's a further refinement in that hostnames can be added to the `xpack.actions.proxyBypassHosts` and `xpack.actions.proxyOnlyHosts` config keys. Only one of these config keys can be used at a time. If the target URL hostname of the HTTP request is listed in the `proxyBypassHosts` list, the proxy won't be used. If the target URL hostname of the HTTP request is **NOT** listed in the `proxyOnlyHosts` list, the proxy won't be used. Depending on the customer's environment, it may be easier to list the hosts to bypass, or easier to list the hosts that should only be proxied, so they can choose either method.

…#95365) (#96491) resolves #92949 This PR adds two new Kibana config keys to further customize when the proxy is used when making HTTP requests. Prior to this PR, if a proxy was set via the `xpack.actions.proxyUrl` config key, all requests would be proxied. Now, there's a further refinement in that hostnames can be added to the `xpack.actions.proxyBypassHosts` and `xpack.actions.proxyOnlyHosts` config keys. Only one of these config keys can be used at a time. If the target URL hostname of the HTTP request is listed in the `proxyBypassHosts` list, the proxy won't be used. If the target URL hostname of the HTTP request is **NOT** listed in the `proxyOnlyHosts` list, the proxy won't be used. Depending on the customer's environment, it may be easier to list the hosts to bypass, or easier to list the hosts that should only be proxied, so they can choose either method.

jagjag added the bug Fixes for quality problems that affect the customer experience label Feb 26, 2021

jagjag changed the title ~~Alerting mail action goes failed due to enabled HTTP proxy in kibana~~ [Alerting] Alerting mail action goes failed due to enabled HTTP proxy in kibana Feb 26, 2021

gmmorris added Feature:Actions Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels Feb 26, 2021

gmmorris changed the title ~~[Alerting] Alerting mail action goes failed due to enabled HTTP proxy in kibana~~ [Actions] email action fails due to enabled HTTP proxy in kibana Feb 26, 2021

pmuellr added this to SDH in Kibana Alerting Mar 2, 2021

pmuellr removed this from SDH in Kibana Alerting Mar 2, 2021

gmmorris removed the bug Fixes for quality problems that affect the customer experience label Mar 2, 2021

mikecote added this to In Progress in Kibana Alerting Mar 2, 2021

mikecote assigned pmuellr Mar 2, 2021

pmuellr mentioned this issue Mar 9, 2021

[actions] allow ssl/tls options to be customized for actions #80120

Closed

pmuellr mentioned this issue Mar 24, 2021

[actions] adds proxyBypassHosts and proxyOnlyHosts Kibana config keys #95365

Merged

9 tasks

pmuellr linked a pull request Mar 25, 2021 that will close this issue

[actions] adds proxyBypassHosts and proxyOnlyHosts Kibana config keys #95365

Merged

9 tasks

This was referenced Mar 25, 2021

[actions] support list of url or host "patterns" to bypass when a proxy is in use #95062

Closed

[Alerts][Actions] The Email connector stopped working after the upgrade Kibana from 7.10.1 to 7.11.1. #94021

Closed

pmuellr closed this as completed in #95365 Apr 7, 2021

Kibana Alerting automation moved this from In Progress to Done (Ordered by most recent) Apr 7, 2021

kobelb added the needs-team Issues missing a team label label Jan 31, 2022

botelastic bot removed the needs-team Issues missing a team label label Jan 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Actions] email action fails due to enabled HTTP proxy in kibana #92949

[Actions] email action fails due to enabled HTTP proxy in kibana #92949

jagjag commented Feb 26, 2021

elasticmachine commented Feb 26, 2021

pmuellr commented Mar 1, 2021

pmuellr commented Mar 2, 2021 •

edited

Loading

gmmorris commented Mar 2, 2021

jagjag commented Mar 4, 2021 •

edited

Loading

pmuellr commented Mar 4, 2021

pmuellr commented Mar 4, 2021

simioa commented Mar 9, 2021

pmuellr commented Mar 9, 2021

pmuellr commented Mar 24, 2021

[Actions] email action fails due to enabled HTTP proxy in kibana #92949

[Actions] email action fails due to enabled HTTP proxy in kibana #92949

Comments

jagjag commented Feb 26, 2021

elasticmachine commented Feb 26, 2021

pmuellr commented Mar 1, 2021

pmuellr commented Mar 2, 2021 • edited Loading

gmmorris commented Mar 2, 2021

jagjag commented Mar 4, 2021 • edited Loading

pmuellr commented Mar 4, 2021

pmuellr commented Mar 4, 2021

simioa commented Mar 9, 2021

pmuellr commented Mar 9, 2021

pmuellr commented Mar 24, 2021

pmuellr commented Mar 2, 2021 •

edited

Loading

jagjag commented Mar 4, 2021 •

edited

Loading