error when enable https dashboard emqx #4047

ilhamderi · 2021-01-22T02:52:41Z

Hi, I need help. I use emqx docker and want to enable https dashboard. when I trying to access the dashboard is blank only white screen, but when I access http dashboard, the dashboard can show up. tell me if I'm wrong with my docker-compose.yml or I need some configuration
Thanks
this my docker-compose.yml

version: '3'

services:

  emqx:
    image: emqx/emqx:4.2.6
    container_name: emqx
    restart: always
    ports:
      - 1883:1883
      - 1885:1885
      - 1884:1884
      - 1886:1886
      - 18083:18083
      - 18084:18084
      - 4369:4369
      - 5369:5369
      - 6369:6369
      - 11883:11883
      - 9540:9540
    volumes:
      - ./ssl:/opt/emqx/etc/certs
      - ./acl.conf:/opt/emqx/etc/acl.conf
    environment:
      - EMQX_NAME=emqx
      - EMQX_NODE__ASYNC__THREADS=32
      - EMQX_NODE__DIST_BUFFER_SIZE=8MB
      - EMQX_NODE__FULLSWEEP_AFTER=1000
      - EMQX_NODE__DIST_NET_TICKTIME=60
      - EMQX_LISTENER__SSL__EXTERNAL=1885
      - EMQX_LISTENER__TCP__EXTERNAL=1883
      - EMQX_LISTENER__WS__EXTERNAL=1884
      - EMQX_LISTENER__WSS__EXTERNAL=1886
      - EMQX_MQTT__MAX_PACKET_SIZE=1MB
      - EMQX_LISTENER__SSL__KEYFILE=/opt/emqx/etc/certs/emqx.key
      - EMQX_LISTENER__SSL__CERTFILE=/opt/emqx/etc/certs/emqx.crt
      - EMQX_LISTENER__SSL__CACERTFILE=etc/certs/emqx.key.chained.crt
      - EMQX_LISTENER__WSS__KEYFILE=/opt/emqx/etc/certs/emqx.key
      - EMQX_LISTENER__WSS__CERTFILE=/opt/emqx/etc/certs/emqx.crt
      - EMQX_LISTENER__WSS__CACERTFILE=etc/certs/emqx.key.chained.crt
      - EMQX_LOADED_PLUGINS="emqx_management | emqx_dashboard | emqx_retainer | emqx_rule_engine | emqx_recon"
      - EMQX_DASHBOARD__LISTENER__HTTP=18083
      - EMQX_DASHBOARD__LISTENER__HTTPS=18084
      - EMQX_DASHBOARD__LISTENER__HTTPS__ACCEPTORS=2
      - EMQX_DASHBOARD__LISTENER__HTTPS__MAX_CLIENTS=10
      - EMQX_DASHBOARD__LISTENER__HTTPS__KEYFILE=/opt/emqx/etc/certs/emqx.key
      - EMQX_DASHBOARD__LISTENER__HTTPS__CERTFILE=/opt/emqx/etc/certs/emqx.crt
      - EMQX_DASHBOARD__LISTENER__HTTPS__CACERTFILE=/opt/emqx/etc/certs/emqx.key.chained.crt
      - EMQX_RETAINED__STORAGE_TYPE=ram
      - EMQX_RETAINED__MAX_RETAINED_MESSAGES=0
      - EMQX_RETAINED__MAX_PAYLOAD_SIZE=256KB
      - EMQX_RETAINED__EXPIRY_INTERVAL=1m

and when I try to access the dashboard (https://mydomain.com:18084), this is log from emqx.log.1

2021-01-22 02:43:52.745 [error]   crasher:
    initial call: cowboy_tls:connection_process/4
    pid: <0.1981.0>
    registered_name: []
    exception error: undefined function cow_http2_machine:get_connection_local_buffer_size/1
      in function  cowboy_http2:maybe_send_data_alarm/3 (cowboy_http2.erl, line 823)
      in call from cowboy_http2:commands/3 (cowboy_http2.erl, line 622)
      in call from cowboy_http2:loop/2 (cowboy_http2.erl, line 247)
    ancestors: [<0.1654.0>,<0.1653.0>,ranch_sup,<0.1403.0>]
    message_queue_len: 1
    messages: [{'EXIT',<0.1982.0>,normal}]
    links: [<0.1654.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 28690
    stack_size: 27
    reductions: 4306
  neighbours:

2021-01-22 02:43:52.746 [error] Ranch listener 'https:dashboard' had connection process started with cowboy_tls:start_link/4 at <0.1981.0> exit with reason: {undef,[{cow_http2_machine,get_connection_local_buffer_size,[{http2_machine,server,#{},normal,undefined,#Ref<0.354825949.3343384578.103752>,#{initial_window_size => 65535},#{},#{header_table_size => 65536,initial_window_size => 6291456,max_concurrent_streams => 1000,max_header_list_size => 262144},15728640,65535,2,1,[{stream,1,<<"GET">>,nofin,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined}],[],[],{state,877,4096,4096,[{61,{<<"accept-language">>,<<"en-US,en;q=0.9">>}},{64,{<<"accept-encoding">>,<<"gzip, deflate, br">>}},{54,{<<"sec-fetch-dest">>,<<"document">>}},{48,{<<"sec-fetch-user">>,<<"?1">>}},{54,{<<"sec-fetch-mode">>,<<"navigate">>}},{50,{<<"sec-fetch-site">>,<<"none">>}},{173,{<<"accept">>,<<"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9">>}},{146,{<<"user-agent">>,<<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36">>}},{58,{<<"upgrade-insecure-requests">>,<<"1">>}},{53,{<<"cache-control">>,<<"no-cache">>}},{46,{<<"pragma">>,<<"no-cache">>}},{70,{<<":authority">>,<<"mydomain.com:18084">>}}]},{state,384,4096,4096,[{44,{<<"server">>,<<"Cowboy">>}},{74,{<<"last-modified">>,<<"Sat, 16 Jan 2021 13:51:03 GMT">>}},{48,{<<"etag">>,<<"\"2429255563\"">>}},{65,{<<"date">>,<<"Fri, 22 Jan 2021 02:43:52 GMT">>}},{53,{<<"content-type">>,<<"text/html">>}},{50,{<<"content-length">>,<<"1130">>}},{50,{<<"accept-ranges">>,<<"bytes">>}}]}}],[]},{cowboy_http2,maybe_send_data_alarm,3,[{file,"cowboy_http2.erl"},{line,823}]},{cowboy_http2,commands,3,[{file,"cowboy_http2.erl"},{line,622}]},{cowboy_http2,loop,2,[{file,"cowboy_http2.erl"},{line,247}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}

2021-01-22 02:43:53.049 [error]   crasher:
    initial call: cowboy_tls:connection_process/4
    pid: <0.1987.0>
    registered_name: []
    exception error: undefined function cow_http2_machine:get_connection_local_buffer_size/1
      in function  cowboy_http2:maybe_send_data_alarm/3 (cowboy_http2.erl, line 823)
      in call from cowboy_http2:commands/3 (cowboy_http2.erl, line 622)
      in call from cowboy_http2:loop/2 (cowboy_http2.erl, line 247)
    ancestors: [<0.1654.0>,<0.1653.0>,ranch_sup,<0.1403.0>]
    message_queue_len: 11
    messages: [{'EXIT',<0.1990.0>,normal},
                  {{<0.1987.0>,3},
                   {response,200,
                       #{<<"accept-ranges">> => <<"bytes">>,
                         <<"content-length">> => <<"560035">>,
                         <<"content-type">> =>
                             [<<"text">>,<<"/">>,<<"css">>,<<>>],
                         <<"date">> => <<"Fri, 22 Jan 2021 02:43:52 GMT">>,
                         <<"etag">> => [34,<<"3795938318">>,34],
                         <<"last-modified">> =>
                             <<"Sat, 16 Jan 2021 13:51:03 GMT">>,
                         <<"server">> => <<"Cowboy">>},
                       {sendfile,0,560035,
                           <<"/opt/emqx/lib/emqx_dashboard-4.2.6/priv/www/static/css/app.1495f134b9420661c25a03fc6be1c155.css">>}}},
                  {'EXIT',<0.1991.0>,normal},
                  {{<0.1987.0>,5},
                   {response,200,
                       #{<<"accept-ranges">> => <<"bytes">>,
                         <<"content-length">> => <<"59563">>,
                         <<"content-type">> =>
                             [<<"text">>,<<"/">>,<<"css">>,<<>>],
                         <<"date">> => <<"Fri, 22 Jan 2021 02:43:52 GMT">>,
                         <<"etag">> => [34,<<"60974109">>,34],
                         <<"last-modified">> =>
                             <<"Sat, 16 Jan 2021 13:51:03 GMT">>,
                         <<"server">> => <<"Cowboy">>},
                       {sendfile,0,59563,
                           <<"/opt/emqx/lib/emqx_dashboard-4.2.6/priv/www/static/css/iconfont.css">>}}},
                  {'EXIT',<0.1992.0>,normal},
                  {{<0.1987.0>,7},
                   {response,200,
                       #{<<"accept-ranges">> => <<"bytes">>,
                         <<"content-length">> => <<"2070">>,
                         <<"content-type">> =>
                             [<<"application">>,<<"/">>,<<"javascript">>,<<>>],
                         <<"date">> => <<"Fri, 22 Jan 2021 02:43:52 GMT">>,
                         <<"etag">> => [34,<<"1148403076">>,34],
                         <<"last-modified">> =>
                             <<"Sat, 16 Jan 2021 13:51:03 GMT">>,
                         <<"server">> => <<"Cowboy">>},
                       {sendfile,0,2070,
                           <<"/opt/emqx/lib/emqx_dashboard-4.2.6/priv/www/static/js/manifest.f72a5c400fac2e53484f.js">>}}},
                  {'EXIT',<0.1993.0>,normal},
                  {{<0.1987.0>,9},
                   {response,200,
                       #{<<"accept-ranges">> => <<"bytes">>,
                         <<"content-length">> => <<"978168">>,
                         <<"content-type">> =>
                             [<<"application">>,<<"/">>,<<"javascript">>,<<>>],
                         <<"date">> => <<"Fri, 22 Jan 2021 02:43:52 GMT">>,
                         <<"etag">> => [34,<<"249154733">>,34],
                         <<"last-modified">> =>
                             <<"Sat, 16 Jan 2021 13:51:03 GMT">>,
                         <<"server">> => <<"Cowboy">>},
                       {sendfile,0,978168,
                           <<"/opt/emqx/lib/emqx_dashboard-4.2.6/priv/www/static/js/vendor.7837b8f015b7d486b74f.js">>}}},
                  {'EXIT',<0.1994.0>,normal},
                  {{<0.1987.0>,11},
                   {response,200,
                       #{<<"accept-ranges">> => <<"bytes">>,
                         <<"content-length">> => <<"66">>,
                         <<"content-type">> =>
                             [<<"application">>,<<"/">>,<<"javascript">>,<<>>],
                         <<"date">> => <<"Fri, 22 Jan 2021 02:43:52 GMT">>,
                         <<"etag">> => [34,<<"9827911">>,34],
                         <<"last-modified">> =>
                             <<"Sat, 16 Jan 2021 13:51:03 GMT">>,
                         <<"server">> => <<"Cowboy">>},
                       {sendfile,0,66,
                           <<"/opt/emqx/lib/emqx_dashboard-4.2.6/priv/www/static/js/env.js">>}}},
                  {'EXIT',<0.1995.0>,normal}]
    links: [<0.1996.0>,<0.1654.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 46422
    stack_size: 27
    reductions: 7404
  neighbours:

2021-01-22 02:43:53.054 [error] Ranch listener 'https:dashboard' had connection process started with cowboy_tls:start_link/4 at <0.1987.0> exit with reason: {undef,[{cow_http2_machine,get_connection_local_buffer_size,[{http2_machine,server,#{},normal,undefined,#Ref<0.354825949.3343384578.103788>,#{initial_window_size => 65535},#{},#{header_table_size => 65536,initial_window_size => 6291456,max_concurrent_streams => 1000,max_header_list_size => 262144},15728640,65535,2,13,[{stream,1,<<"GET">>,nofin,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined},{stream,3,<<"GET">>,idle,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined},{stream,5,<<"GET">>,idle,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined},{stream,7,<<"GET">>,idle,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined},{stream,9,<<"GET">>,idle,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined},{stream,11,<<"GET">>,idle,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined},{stream,13,<<"GET">>,idle,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined}],[],[],{state,826,4096,4096,[{52,{<<"sec-fetch-dest">>,<<"script">>}},{41,{<<"accept">>,<<"*/*">>}},{61,{<<"accept-language">>,<<"en-US,en;q=0.9">>}},{64,{<<"accept-encoding">>,<<"gzip, deflate, br">>}},{76,{<<"referer">>,<<"https://mydomain.com:18084/">>}},{51,{<<"sec-fetch-dest">>,<<"style">>}},{53,{<<"sec-fetch-mode">>,<<"no-cors">>}},{57,{<<"sec-fetch-site">>,<<"same-origin">>}},{56,{<<"accept">>,<<"text/css,*/*;q=0.1">>}},{146,{<<"user-agent">>,<<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36">>}},{53,{<<"cache-control">>,<<"no-cache">>}},{46,{<<"pragma">>,<<"no-cache">>}},{70,{<<":authority">>,<<"mydomain.com:18084">>}}]},{state,384,4096,4096,[{44,{<<"server">>,<<"Cowboy">>}},{74,{<<"last-modified">>,<<"Sat, 16 Jan 2021 13:51:03 GMT">>}},{48,{<<"etag">>,<<"\"1235029800\"">>}},{65,{<<"date">>,<<"Fri, 22 Jan 2021 02:43:52 GMT">>}},{52,{<<"content-type">>,<<"text/css">>}},{51,{<<"content-length">>,<<"31000">>}},{50,{<<"accept-ranges">>,<<"bytes">>}}]}}],[]},{cowboy_http2,maybe_send_data_alarm,3,[{file,"cowboy_http2.erl"},{line,823}]},{cowboy_http2,commands,3,[{file,"cowboy_http2.erl"},{line,622}]},{cowboy_http2,loop,2,[{file,"cowboy_http2.erl"},{line,247}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}

The text was updated successfully, but these errors were encountered:

enrahmad · 2021-01-24T05:36:04Z

Can you try change to version v4.2.2 ? Hope it works.

I also facing the same error when use v4.2.6, it works using http only, here logs I get

(6f3844deb39b@172.29.0.2)1> 2021-01-24 04:58:42.940 [error] Ranch listener 'https:dashboard' had connection process started with cowboy_tls:start_link/4 at <0.2418.0> exit with reason: {undef,[{cow_http2_machine,get_connection_local_buffer_size,[{http2_machine,server,#{},normal,undefined,#Ref<0.1950698705.2199650305.195207>,#{initial_window_size => 65535},#{},#{header_table_size => 65536,initial_window_size => 6291456,max_concurrent_streams => 1000,max_header_list_size => 262144},15728640,65535,2,9,[{stream,1,<<"GET">>,idle,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined},{stream,3,<<"GET">>,nofin,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined},{stream,5,<<"GET">>,idle,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined},{stream,7,<<"GET">>,idle,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined},{stream,9,<<"GET">>,idle,6291456,{[],[]},0,undefined,fin,65535,0,0,undefined}],[],[],{state,827,4096,4096,[{52,{<<"sec-fetch-dest">>,<<"script">>}},{41,{<<"accept">>,<<"*/*">>}},{90,{<<"cookie">>,<<"__cfduid=dc0cd13c24f2fe6f742e471a6d12be57c1610938396">>}},{70,{<<"accept-language">>,<<"en-US,en;q=0.9,id;q=0.8">>}},{64,{<<"accept-encoding">>,<<"gzip, deflate, br">>}},{71,{<<"referer">>,<<"https://mydomain.com:18084/">>}},{51,{<<"sec-fetch-dest">>,<<"style">>}},{53,{<<"sec-fetch-mode">>,<<"no-cors">>}},{57,{<<"sec-fetch-site">>,<<"same-origin">>}},{56,{<<"accept">>,<<"text/css,*/*;q=0.1">>}},{157,{<<"user-agent">>,<<"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36">>}},{65,{<<":authority">>,<<"mydomain.com:18084">>}}]},{state,397,4096,4096,[{44,{<<"server">>,<<"Cowboy">>}},{74,{<<"last-modified">>,<<"Sat, 16 Jan 2021 13:51:03 GMT">>}},{48,{<<"etag">>,<<"\"1148403076\"">>}},{65,{<<"date">>,<<"Sun, 24 Jan 2021 04:58:42 GMT">>}},{66,{<<"content-type">>,<<"application/javascript">>}},{50,{<<"content-length">>,<<"2070">>}},{50,{<<"accept-ranges">>,<<"bytes">>}}]}}],[]},{cowboy_http2,maybe_send_data_alarm,3,[{file,"cowboy_http2.erl"},{line,823}]},{cowboy_http2,commands,3,[{file,"cowboy_http2.erl"},{line,622}]},{cowboy_http2,loop,2,[{file,"cowboy_http2.erl"},{line,247}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}

And when you check console at web browser, you will get some files js can not loaded.

env.js
app.xxx.js
iconfont.css
etc

ilhamderi · 2021-01-25T05:21:54Z

I have changed to version v4.2.2 and it's work for me thanks @enrahmad

GJean · 2021-02-07T23:50:29Z

Hello,

I have same error with v4.2.7 in Ubuntu 20.10.

enrahmad · 2021-02-07T23:56:52Z

Not sure, but I think it's bug, not related with your OS.

…

On Mon, Feb 8, 2021, 6:50 AM GJean ***@***.***> wrote: Hello, I have same error with v4.2.7 in Ubuntu 20.10. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4047 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACQGLBK4TJRJ6VBOBVKMI23S54RNDANCNFSM4WN2XTQQ> .

caciuto · 2021-04-20T16:20:55Z

facing the same with v4.2.11 with Debian-10
Certificates used have been generated with Letsencrypt. Don't know if it is relevant...

k32 · 2021-04-30T14:35:28Z

It looks like a packaging problem. I see that EMQX v4.2.11 uses cowboy 2.8.1, which depends on cowlib 2.9.1 according to the app file. While EMQ X docker container ships with cowlib 2.7.0

Correction: EMQ X uses a forked cowboy with its own versioning.

zmstone · 2021-04-30T15:20:21Z

it looks like emqx_auth_http got its priority in dependency resolution.

emqx/lib/emqtt/rebar.config:9:10:        {cowlib, {git, "https://github.com/ninenines/cowlib", {tag, "2.6.0"}}},
emqx/lib/gun/rebar.config:2:6:    {cowlib,{git,"https://github.com/ninenines/cowlib",{tag, "2.8.0"}}}
emqx/lib/emqx_auth_http/rebar.config:2:4: [{cowlib, {git, "https://github.com/ninenines/cowlib", {tag, "2.7.0"}}},
emqx/lib/cowboy/rebar.config:5:7:     {cowlib, {git, "https://github.com/ninenines/cowlib", {tag, "2.8.0"}}}

k32 · 2021-04-30T15:24:03Z

Cowlib commit that introduced the missing function: ninenines/cowlib@0a1fb6c (tags: 2.10.1 2.10.0 2.9.1 2.9.0 2.8.0)

zmstone · 2021-04-30T15:27:48Z

sorry, I did not set the correct environment variable.
these are the right dependencies for v4.2.11

===> Fetching cowboy (from {git,"https://github.com/emqx/cowboy",{tag,"2.7.1"}})
===> Fetching cowlib (from {git,"https://github.com/ninenines/cowlib",{tag,"2.7.0"}})

zmstone · 2021-04-30T15:29:35Z

and it's 2.8.1 app version at 2.7.1 tag.
https://github.com/emqx/cowboy/blob/2.7.1/src/cowboy.app.src#L3
what a mess!

k32 · 2021-04-30T16:50:32Z

This PR should fix the cowlib version: https://github.com/emqx/emqx-auth-http/pull/260/files
It should be released in 4.2.12

zmstone · 2021-04-30T16:54:41Z

Thank you @k32
We'll see if we can make a new release on Monday.

nishant-vvdn · 2021-05-11T07:51:35Z

Hi, @k32 and @zmstone

Is this issue fixed in the latest release?

k32 · 2021-05-11T08:19:32Z

Yes, 4.3.0 includes the correct version of cowlib.

ilhamderi added the Support label Jan 22, 2021

ilhamderi assigned tigercl Jan 22, 2021

ilhamderi changed the title ~~error enable https dashboard emqx~~ error when enable https dashboard emqx Jan 22, 2021

ilhamderi closed this as completed Jan 25, 2021

k32 reopened this Apr 30, 2021

k32 self-assigned this Apr 30, 2021

k32 closed this as completed May 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error when enable https dashboard emqx #4047

error when enable https dashboard emqx #4047

ilhamderi commented Jan 22, 2021 •

edited

enrahmad commented Jan 24, 2021

ilhamderi commented Jan 25, 2021

GJean commented Feb 7, 2021

enrahmad commented Feb 7, 2021 via email

caciuto commented Apr 20, 2021

k32 commented Apr 30, 2021 •

edited

zmstone commented Apr 30, 2021

k32 commented Apr 30, 2021 •

edited

zmstone commented Apr 30, 2021

zmstone commented Apr 30, 2021

k32 commented Apr 30, 2021

zmstone commented Apr 30, 2021

nishant-vvdn commented May 11, 2021

k32 commented May 11, 2021

error when enable https dashboard emqx #4047

error when enable https dashboard emqx #4047

Comments

ilhamderi commented Jan 22, 2021 • edited

enrahmad commented Jan 24, 2021

ilhamderi commented Jan 25, 2021

GJean commented Feb 7, 2021

enrahmad commented Feb 7, 2021 via email

caciuto commented Apr 20, 2021

k32 commented Apr 30, 2021 • edited

zmstone commented Apr 30, 2021

k32 commented Apr 30, 2021 • edited

zmstone commented Apr 30, 2021

zmstone commented Apr 30, 2021

k32 commented Apr 30, 2021

zmstone commented Apr 30, 2021

nishant-vvdn commented May 11, 2021

k32 commented May 11, 2021

ilhamderi commented Jan 22, 2021 •

edited

k32 commented Apr 30, 2021 •

edited

k32 commented Apr 30, 2021 •

edited