Mattermost webhook plugin post error - dail tcp i/o error

[tejaskarelia17]

Summary

I am unable to make any post request from a plugin. dail tcp IP i/o error

Steps to reproduce

I have installed Mattermost 5.13.2 on a DMZ (192.168.x.x:8067) behind a Firewall using NAT. I am able to use Mattermost as expected from internet using my public IP/hostname as well as internal.
I installed a plugin(RemindBot) but none of the buttons work after a reminder is sent.

I changed the config file to reflect the following:

"EnableInsecureOutgoingConnections": true,
"AllowedUntrustedInternalConnections": “192.168.x.x”,

I still am unable to make it work.
Observed behavior

Following is the error that is generated:

{“level”:“error”,“ts”:1564767183.8125181,“caller”:“mlog/log.go:172”,“msg”:“Action integration error”,“path”:"/api/v4/posts/nrct1bbkdpyqtm41jdfyanhmaa/actions/feqngh9b1jyetpqianhwwcmkmo",“request_id”:“4ypkej9zdpgmpmpcrq96jfxkuw”,“ip_addr”:“10.190.x.x”,“user_id”:“c859kpm4e38odqttj7y9cz367r”,“method”:“POST”,“err_where”:“DoActionRequest”,“http_code”:400,“err_details”:“err=Post https://{hostname}:8067/plugins/com.github.scottleedavis.mattermost-plugin-remind/delete: dial tcp PUBLIC_IP:8067: i/o timeout”}

[scottleedavis]

@tejaskarelia17 is SiteURL valid? If so I suspect a firewall issue with an i/o timeout error. If another plugin say, matterpoll didn't work this is likely the issue.

[scottleedavis]

@tejaskarelia17 This plugin will update to use relative paths for mattermost release 5.14

[scottleedavis]

following the nps plugin as an example mattermost/mattermost-plugin-nps#45

[tejaskarelia17]

@scottleedavis Thanks 😃

[johnthompson365]

Hi Scott - I have a customer who loves the plugin but is getting a similar error, without the explicit timeout. Is there any further troubleshooting or is it just best to wait for v4.4? Thanks.

{"level":"error","ts":1565912527.5476964,"caller":"mlog/log.go:172","msg":"Action integration error","path":"/api/v4/posts/j8zwbedfxfytpg4hf3uhu8onza/actions/orfrnyxw93rgjb4bhjxc8dzrhy","request_id":"szwyfomzdbgc5bwx6fyhgfxgow","ip_addr":"192.168.x.x","user_id":"5frpib9ss3ruwey4ip6zxo49so","method":"POST","err_where":"DoActionRequest","http_code":400,"err_details":"status=400"}

[scottleedavis]

Howdy John,

In this case lets wait to see if 0.4.4 resolves it, which should be released friday (following a 5.14 release requirement)

[scottleedavis]

Hello! v0.4.4 has been released!
https://github.com/scottleedavis/mattermost-plugin-remind/releases/tag/0.4.4

This should solve this issue @tejaskarelia17 & @johnthompson365 . Let me know if it doesn't! (This version requires mattermost v5.14)

[johnthompson365]

Thanks for the update Scott! Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

…

________________________________ From: scott lee davis <notifications@github.com> Sent: Monday, August 19, 2019 7:56:37 PM To: scottleedavis/mattermost-plugin-remind <mattermost-plugin-remind@noreply.github.com> Cc: John Thompson <john@thompson365.com>; Mention <mention@noreply.github.com> Subject: Re: [scottleedavis/mattermost-plugin-remind] Mattermost webhook plugin post error - dail tcp i/o error (#148) Hello! v0.4.4 has been released! https://github.com/scottleedavis/mattermost-plugin-remind/releases/tag/0.4.4 This should solve this issue @tejaskarelia17<https://github.com/tejaskarelia17> & @johnthompson365<https://github.com/johnthompson365> . Let me know if it doesn't! (This version requires mattermost v5.14) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#148?email_source=notifications&email_token=AK5WVPC57CVXHRO37THZKGTQFNMOLA5CNFSM4IKCJEVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4U4R6Q#issuecomment-522832122>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK5WVPAHGVXTSP5PAVCUG4TQFNMOLANCNFSM4IKCJEVA>.

[tejaskarelia17]

Hi @scottleedavis,
Thanks for the update. I updated mattermost to 5.14.0 and Remind bot 0.4.4. But I am getting the same error.
Is there a way I can check whether the error I am getting is due to the SiteURL parameter?

[johnthompson365]

Thanks for re-opening this, our customer is reporting the same issue with v4.4 as well.

[scottleedavis]

@tejaskarelia17 would it be possible to test another plugin with interactive buttons? e.g. https://github.com/matterpoll/matterpoll

If those buttons work that narrows it down to this plugin. If they don't, we can discuss more on the user voice post to see if mattermost staff can help.

[johnthompson365]

Hi Scott, we tried your suggestion of using Matterpoll and we got the same error. So doesn't look to be plugin related. I assume you can't recreate the issue on your v5.14 server?

[scottleedavis]

I cant recreate it, though I will test with a 5.14 docker image this eve to cover that base.

[johnthompson365]

Thanks Scott. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

…

________________________________ From: scott lee davis <notifications@github.com> Sent: Thursday, August 22, 2019 7:06:13 PM To: scottleedavis/mattermost-plugin-remind <mattermost-plugin-remind@noreply.github.com> Cc: John Thompson <john@thompson365.com>; Mention <mention@noreply.github.com> Subject: Re: [scottleedavis/mattermost-plugin-remind] Mattermost webhook plugin post error - dail tcp i/o error (#148) I cant recreate it, though I will test with a 5.14 docker image this eve to cover that base. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#148?email_source=notifications&email_token=AK5WVPD24FV4JS3UCHGZJ6DQF5AZLA5CNFSM4IKCJEVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4642IY#issuecomment-524143907>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK5WVPDFEKKA5HRKWD47NWLQF5AZLANCNFSM4IKCJEVA>.

[scottleedavis]

@johnthompson365 @tejaskarelia17 I tested the following this eve.

1. docker image of 5.14 => This works without error.
2. dev mattermost-server up-to-date master => This works without error.
3. 5.14 debian install, running in GCE. => This does not work and gives a very similar, but not exact same error as reported

{"level":"error","ts":1566534094.7552865,"caller":"mlog/log.go:173","msg":"Action integration error","path":"/api/v4/posts/nterq7nrm7nyzq8mf1ugsgwhph/actions/q9o3rxsdxbbninw7juzu34q88c","request_
id":"ro65a673ipy37bb3r4tfwus1xo","ip_addr":"IP_ADDR","user_id":"y8e5yubqofgm7f69sthoet3goy","method":"POST","err_where":"DoActionRequest","http_code":400,"err_details":"err=Post http://IP_ADDR/plugins/com.github.scottleedavis.mattermost-plugin-remind/delete/ephemeral: dial tcp IP_ADDR:80: connect: connection refused"}

(IP_ADDR replaced actual ip)

The AllowedUntrustedInternalConnections was also set to IP_ADDR.

A discussion thread created by john is here

The mattermost forum thread is here

@lieut-data would you like me to add my findings/status on the mattermost forum thread, or best to discuss here?

[johnthompson365]

Thanks for investigating Scott. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

…

________________________________ From: scott lee davis <notifications@github.com> Sent: Thursday, August 22, 2019 9:43:40 PM To: scottleedavis/mattermost-plugin-remind <mattermost-plugin-remind@noreply.github.com> Cc: John Thompson <john@thompson365.com>; Mention <mention@noreply.github.com> Subject: Re: [scottleedavis/mattermost-plugin-remind] Mattermost webhook plugin post error - dail tcp i/o error (#148) @johnthompson365<https://github.com/johnthompson365> @tejaskarelia17<https://github.com/tejaskarelia17> I tested the following this eve. 1. docker image of 5.14 => This works without error. 2. dev mattermost-server up-to-date master => This works without error. 3. 5.14 debian install<https://docs.mattermost.com/install/install-debian.html>, running in GCE. => This does not work and gives a very similar, but not exact same error as reported {"level":"error","ts":1566534094.7552865,"caller":"mlog/log.go:173","msg":"Action integration error","path":"/api/v4/posts/nterq7nrm7nyzq8mf1ugsgwhph/actions/q9o3rxsdxbbninw7juzu34q88c","request_ id":"ro65a673ipy37bb3r4tfwus1xo","ip_addr":"IP_ADDR","user_id":"y8e5yubqofgm7f69sthoet3goy","method":"POST","err_where":"DoActionRequest","http_code":400,"err_details":"err=Post http://IP_ADDR/plugins/com.github.scottleedavis.mattermost-plugin-remind/delete/ephemeral: dial tcp IP_ADDR:80: connect: connection refused"} (IP_ADDR replaced actual ip) The AllowedUntrustedInternalConnections was also set to IP_ADDR. A discussion thread created by john is here<https://community.mattermost.com/core/pl/zx3kkudg5bf1dc9m6tapqzrnrh> The mattermost forum thread is here<https://forum.mattermost.org/t/mattermost-webhook-plugin-post-error-dail-tcp-i-o-error/7823> @lieut-data<https://github.com/lieut-data> would you like me to add my findings/status on the mattermost forum thread, or best to discuss here? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#148?email_source=notifications&email_token=AK5WVPACVDT4XPO5Q7CMPD3QF5THZA5CNFSM4IKCJEVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD47DHSQ#issuecomment-524170186>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK5WVPCUE6DMW7IDCG7XFBDQF5THZANCNFSM4IKCJEVA>.

[lieut-data]

@scottleedavis, thanks for the observations! I've replied to https://community-daily.mattermost.com/core/pl/zx3kkudg5bf1dc9m6tapqzrnrh with a theory -- let me know what you think :)

[scottleedavis]

I resonate with the theory @lieut-data , though did get the 404 page not found when running the curl POST test. I am hopeful mattermost/mattermost#11964 will solve this.

[scottleedavis]

@tejaskarelia17 Could it be related to siteurl needing to be https? That solved the problem @johnthompson365 was looking into.

[scottleedavis]

also @tejaskarelia17 , here is a workaround by @JtheBAB

My solution was to create on the MM server in /etc/hosts a dns entry that points the fqdn to the MM server instead of the reverse proxy. Then the mm server finds itself.

Do either of these two solve the issue you are seeing @tejaskarelia17 ?

[tejaskarelia17]

@tejaskarelia17 Could it be related to siteurl needing to be https? That solved the problem @johnthompson365 was looking into.
@scottleedavis I have my site url set to https:. I am going to try the workaround suggested by @JtheBAB and update the post asap. Thanks.

[tejaskarelia17]

The work around suggested by @JtheBAB works 😃 👍
Thanks @JtheBAB, @scottleedavis and everyone for all the help.

[scottleedavis]

Excellent news! Thank you all!