acme/autocert: missing certificate - remote installation public IP

[ecities]

Summary

The stack (most current version) - returns missing certificate from acme.
Does LetsEncrypt certification require some additional tasks beyond specified in 'certification' chapter of getting started manual?.
Earlier on different machine I was testing certbot (https://subdomain.example.com) was opened succesfully in https on apache web server. The stack return the same errors.
...

Steps to Reproduce

1. Installation from getting started debian 9 stretch with script on public IP with subdomain.example.com. Used script from Problems with management remote VPS server + static IP (ttn-lw-cli + webgui) #1723

2. Stack seems working ok (no errors)

3. I'm trying to open https://subdomain.example.com:8885/, It returns
   in web browser:
   Error code: SSL_ERROR_INTERNAL_ERROR_ALERT
   in console:

stack_1      |   INFO Listening for connections                address=:1884 namespace=grpc protocol=gRPC
stack_1      |   INFO Listening for connections                address=:8884 namespace=grpc protocol=gRPC/tls
stack_1      |   INFO Listening for connections                address=:1885 namespace=web protocol=Web
stack_1      |   INFO Listening for connections                address=:8885 namespace=web protocol=Web/tls
stack_1      |   INFO Listening for connections                address=:8886 namespace=interop protocol=Interop/tls
stack_1      | 2019/12/16 08:27:25 http: TLS handshake error from 178.42.42.42:41088: acme/autocert: unable to satisfy "https://acme-v02.api.letsencrypt.org/acme/authz-v3/1757836769" for domain "lorawan.hiiq.systems": no viable challenge type found
stack_1      | 2019/12/16 08:27:25 http: TLS handshake error from 178.42.42.42:41090: acme/autocert: missing certificate

What do you see now?

root@debian9:~# docker-compose up
Creating network "root_default" with the default driver
Creating root_redis_1 ... done
Creating root_cockroach_1 ... done
Creating root_stack_1 ... done
Attaching to root_redis_1, root_cockroach_1, root_stack_1
redis_1 | 1:C 16 Dec 2019 08:26:42.402 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 16 Dec 2019 08:26:42.402 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 16 Dec 2019 08:26:42.402 # Configuration loaded
redis_1 | 1:M 16 Dec 2019 08:26:42.404 * Running mode=standalone, port=6379.
redis_1 | 1:M 16 Dec 2019 08:26:42.404 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 16 Dec 2019 08:26:42.404 # Server initialized
redis_1 | 1:M 16 Dec 2019 08:26:42.404 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 16 Dec 2019 08:26:42.404 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | 1:M 16 Dec 2019 08:26:42.404 * DB loaded from append only file: 0.000 seconds
redis_1 | 1:M 16 Dec 2019 08:26:42.404 * Ready to accept connections
cockroach_1 | *
cockroach_1 | * WARNING: RUNNING IN INSECURE MODE!
cockroach_1 | *
cockroach_1 | * - Your cluster is open for any client that can access .
cockroach_1 | * - Any user, even root, can log in without providing a password.
cockroach_1 | * - Any user, connecting as root, can read or write any data in your cluster.
cockroach_1 | * - There is no network encryption nor authentication, and thus no confidentiality.
cockroach_1 | *
cockroach_1 | * Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v19.2/secure-a-cluster.html
cockroach_1 | *
cockroach_1 | *
cockroach_1 | * WARNING: running 'cockroach start' without --join is deprecated.
cockroach_1 | * Consider using 'cockroach start-single-node' or 'cockroach init' instead.
cockroach_1 | *
cockroach_1 | *
cockroach_1 | * WARNING: neither --listen-addr nor --advertise-addr was specified.
cockroach_1 | * The server will advertise "a058797f72d0" to other nodes, is this routable?
cockroach_1 | *
cockroach_1 | * Consider using:
cockroach_1 | * - for local-only servers: --listen-addr=localhost
cockroach_1 | * - for multi-node clusters: --advertise-addr=<host/IP addr>
cockroach_1 | *
cockroach_1 | *
stack_1 | INFO Setting up core component
stack_1 | INFO Setting up Identity Server
cockroach_1 | CockroachDB node starting at 2019-12-16 08:26:45.68460423 +0000 UTC (took 2.9s)
cockroach_1 | build: CCL v19.2.1 @ 2019/11/18 23:23:55 (go1.12.12)
cockroach_1 | webui: http://a058797f72d0:26256
cockroach_1 | sql: postgresql://root@a058797f72d0:26257?sslmode=disable
cockroach_1 | RPC client flags: /cockroach/cockroach --host=a058797f72d0:26257 --insecure
cockroach_1 | logs: /cockroach/cockroach-data/logs
cockroach_1 | temp dir: /cockroach/cockroach-data/cockroach-temp829040859
cockroach_1 | external I/O path: /cockroach/cockroach-data/extern
cockroach_1 | store[0]: path=/cockroach/cockroach-data
cockroach_1 | status: restarted pre-existing node
cockroach_1 | clusterID: a5c5638e-e3c3-4ae0-9d85-b35d30299f90
cockroach_1 | nodeID: 1
stack_1 | INFO Setting up Gateway Server
stack_1 | INFO Setting up Network Server
stack_1 | INFO Setting up Application Server
stack_1 | INFO Setting up Join Server
stack_1 | INFO Setting up Console
stack_1 | INFO Setting up Device Template Converter
stack_1 | INFO Setting up QR Code Generator
stack_1 | INFO Starting...
stack_1 | WARN No cluster key configured, generated a random one key=44ef62694c7c1bdc8fc5d220fe62e3dccf9cd596355628825ddb718c9482e910
stack_1 | INFO Listening for connections address=:1884 namespace=grpc protocol=gRPC
stack_1 | INFO Listening for connections address=:8884 namespace=grpc protocol=gRPC/tls
stack_1 | INFO Listening for connections address=:1885 namespace=web protocol=Web
stack_1 | INFO Listening for connections address=:8885 namespace=web protocol=Web/tls
stack_1 | INFO Listening for connections address=:8886 namespace=interop protocol=Interop/tls
stack_1 | 2019/12/16 08:27:25 http: TLS handshake error from 178.42.42.42:41088: acme/autocert: unable to satisfy "https://acme-v02.api.letsencrypt.org/acme/authz-v3/1757836769" for domain "lorawan.hiiq.systems": no viable challenge type found
stack_1 | 2019/12/16 08:27:25 http: TLS handshake error from 178.42.42.42:41090: acme/autocert: missing certificate

...

What do you want to see instead?

...

Environment

debian 9, on PC, FF
...

How do you propose to implement this?

If the letsencrypt certificate need some additional action, could you please be more specific in documentation or link relevant documentation not main page of letsencrypt.
...

Can you do this yourself and submit a Pull Request?

...

[KrishnaIyer]

According to the error here: https://acme-v02.api.letsencrypt.org/acme/authz-v3/1757836769
It seems like it's trying a DNS validation.
Are you trying to fetch wildcard certificates for your sub-domains? Specifically, what's the value of tls.acme.hosts and tls.acme.default-host?

Also, is http://your-domain:80 accessible?

[ecities]

Hi thanks for reply: I can send you whole logs and 'real' settings if you will send me your email to 'b i u r o (at) i s y s . p l'
mysubdomain (replaced subdomain set in dns and targeted to public IP)
also domain, *.domain are targeted to the same public IP
all keys are masquaraded

http://subdomain/ - shows 'non configured debian page from apache'
https://subdomain/ - is not configured in apache (should it be? or is apache independant? - i tested earlier on other machine with apache configured and workiing succesfully https://subdomain:443/ but TTN throws the same errors)
https://subdomain:8885/ - returns certificate missing errors
both tls.acme.hosts are set to subdomain

We also had error:
TLS handshake error from (station ip) acme:error:rateLimited error creating new order to many failed authorisations recently

docker-compose.yml
`version: "3.7"

services:

cockroach:
image: 'cockroachdb/cockroach:latest' # TODO: Tag for production.
command: 'start --http-port 26256 --insecure'
restart: 'unless-stopped'
volumes:
- './data/cockroach:/cockroach/cockroach-data'

redis:
image: 'redis:latest' # TODO: Tag for production.
command: 'redis-server --appendonly yes'
restart: 'unless-stopped'
volumes:
- './data/redis:/data'

stack:
image: 'thethingsnetwork/lorawan-stack:latest' # TODO: Tag for production.
entrypoint: 'ttn-lw-stack'
command: 'start'
restart: 'unless-stopped'
depends_on:
- 'cockroach'
- 'redis'
volumes:
- './acme:/var/lib/acme'
- './data/blob:/srv/ttn-lorawan/public/blob'
ports:

- '80:1885'

- '443:8885'

  - '1882:1882'
  - '8882:8882'
  - '1883:1883'
  - '8883:8883'
  - '1884:1884'
  - '8884:8884'
  - '8885:8885'
  - '8886:8886'
  - '1887:1887'
  - '8887:8887'
  - '1700:1700/udp'
env_file: '.env'`

.env file:

TTN_LW_IS_DATABASE_URI=postgres://root@cockroach:26257/ttn_lorawan?sslmode=disable
TTN_LW_REDIS_ADDRESS=redis:6379

TTN_LW_CACHE_SERVICE=redis
TTN_LW_CACHE_REDIS_ADDRESS=redis:6379

TTN_LW_EVENTS_BACKEND=redis
TTN_LW_EVENTS_REDIS_ADDRESS=redis:6379

TTN_LW_TLS_SOURCE=acme
TTN_LW_TLS_ACME_DIR=/var/lib/acme
TTN_LW_TLS_ACME_EMAIL=is@mydomain
TTN_LW_TLS_ACME_HOSTS=mysubdomain
TTN_LW_TLS_ACME_DEFAULT_HOST=mysubdomain

TTN_LW_HTTP_COOKIE_HASH_KEY=4d9630f63d114b34fb77f76f999f507e2812f6af0d435f60d63c3a68f4f2f62efa82b742403766b654fefd091d8cbf8a1d88386bfe110082e15
TTN_LW_HTTP_COOKIE_BLOCK_KEY=aa2e623027dc1867469101a6dcc581b66063b555dff012488d7abbe86
TTN_LW_HTTP_METRICS_PASSWORD=7bbf96815fb24e7c17d7752119
TTN_LW_HTTP_PPROF_PASSWORD=7bbf96815fb24e7c17d7752119

TTN_LW_IS_EMAIL_SENDER_NAME=The Things Stack
TTN_LW_IS_EMAIL_SENDER_ADDRESS=noreply@mysubdomain
TTN_LW_IS_EMAIL_NETWORK_CONSOLE_URL=https://mysubdomain/console
TTN_LW_IS_EMAIL_NETWORK_IDENTITY_SERVER_URL=https://mysubdomain/oauth

# Intentionally omitting email provider config

TTN_LW_IS_OAUTH_UI_CANONICAL_URL=https://mysubdomain/oauth
TTN_LW_IS_OAUTH_UI_IS_BASE_URL=https://mysubdomain/api/v3

TTN_LW_CONSOLE_OAUTH_AUTHORIZE_URL=https://mysubdomain/oauth/authorize
TTN_LW_CONSOLE_OAUTH_TOKEN_URL=https://mysubdomain/oauth/token

TTN_LW_CONSOLE_UI_CANONICAL_URL=https://mysubdomain/console
TTN_LW_CONSOLE_UI_AS_BASE_URL=https://mysubdomain/api/v3
TTN_LW_CONSOLE_UI_GS_BASE_URL=https://mysubdomain/api/v3
TTN_LW_CONSOLE_UI_IS_BASE_URL=https://mysubdomain/api/v3
TTN_LW_CONSOLE_UI_JS_BASE_URL=https://mysubdomain/api/v3
TTN_LW_CONSOLE_UI_NS_BASE_URL=https://mysubdomain/api/v3
TTN_LW_CONSOLE_UI_EDTC_BASE_URL=https://mysubdomain/api/v3
TTN_LW_CONSOLE_UI_QRG_BASE_URL=https://mysubdomain/api/v3

TTN_LW_CONSOLE_OAUTH_CLIENT_ID=console
TTN_LW_CONSOLE_OAUTH_CLIENT_SECRET=3qqqqqqqqqqqqqqqqqqqqqqqqqqc

TTN_LW_GS_MQTT_V2_PUBLIC_ADDRESS=mysubdomain:1881
TTN_LW_GS_MQTT_V2_PUBLIC_TLS_ADDRESS=mysubdomain:8881

TTN_LW_GS_MQTT_PUBLIC_ADDRESS=mysubdomain:1882
TTN_LW_GS_MQTT_PUBLIC_TLS_ADDRESS=mysubdomain:8882

TTN_LW_AS_MQTT_PUBLIC_ADDRESS=mysubdomain:1883
TTN_LW_AS_MQTT_PUBLIC_TLS_ADDRESS=mysubdomain:8883

TTN_LW_GCS_BASIC_STATION_DEFAULT_LNS_URI=wss://lmysubdomain:8887
TTN_LW_GCS_THE_THINGS_GATEWAY_DEFAULT_MQTT_SERVER=mqtts://mysubdomain:8881

[ecities]

I did some tests:
disable apache : service apache2 stop
enable ports (removing hash) 80,443 in docker compose file
docker-compose down
docker-compose pull
docker-compose up

and it seams that console starts to work on standard ports: http://subdomain/ https://subdomain/

However, I think it shouldn't colide with other subdomains, main-domain and apache and potentially other domains on the same hosts (multi-site/multi-domain) on VPS or dedicated servers.

Do you have some docs which gives some aproaches to valid configuration for this issues (apache/multidomain/multisite)?

[KrishnaIyer]

Ok. So this still doesn't answer my questions.

a. If you are trying to retrieve a wildcard certificate, then that won't work since Let's Encrypt only supports DNS validation for those. Please read further here.
b. The stack ACME client requires that http://yourdomain:80 be accessible via the internet. That's needed to actually validate your domain. If you have additional sub-domains configured as ACME Hosts, then each of those subdomains (Ex: http://subdomain1:80, http://subdomain2:80 ...) must be externally accessible.

Since this issue is about DNS and Proxy(apache) configuration on public servers and it doesn't fall under the scope of the stack ACME, I'm closing this.

Please feel free to open a Documentation Request Issue/ PR if there needs to be something added there.

[ecities]

Hi, I 've remove wildcard and it's not an issue.
It seams to be apache + ttn stack on the same machine issue.
Its very common situation, if somebody wants to put TTN stack and some DB/web app on the same device.

When apache is active (listening on 80, 443) port configuration - ttn stack do not allow to enable 80, 443 ports (80,443 must be # in docker-compose.yml - ports 1885, 8885 are used) . In this (docker-compose do not get the letsencrypt certificate).
`
#docker-compose.yml file
ports:

- '80:1885'

- '443:8885'

  - '1882:1882'
  - '8882:8882'
  - '1883:1883'
  - '8883:8883'
  - '1884:1884'
  - '8884:8884'
  - '1885:1885'
  - '8885:8885'
  - '8886:8886'
  - '1887:1887'
  - '8887:8887'
  - '1700:1700/udp'
env_file: '.env'

`

`

.env file moved port to 8885 instead 443

TTN_LW_IS_EMAIL_NETWORK_CONSOLE_URL=https://subdomain.example.com**:8885**/console
TTN_LW_IS_EMAIL_NETWORK_IDENTITY_SERVER_URL=https://subdomain.example.com**:8885**/oauth

Intentionally omitting email provider config

TTN_LW_IS_OAUTH_UI_CANONICAL_URL=https://subdomain.example.com**:8885**/oauth
TTN_LW_IS_OAUTH_UI_IS_BASE_URL=https://subdomain.example.com**:8885**/api/v3

TTN_LW_CONSOLE_OAUTH_AUTHORIZE_URL=https://subdomain.example.com**:8885**/oauth/authorize
TTN_LW_CONSOLE_OAUTH_TOKEN_URL=https://subdomain.example.com**:8885**/oauth/token

TTN_LW_CONSOLE_UI_CANONICAL_URL=https://subdomain.example.com**:8885**/console
TTN_LW_CONSOLE_UI_AS_BASE_URL=https://subdomain.example.com**:8885**/api/v3
TTN_LW_CONSOLE_UI_GS_BASE_URL=https://subdomain.example.com**:8885**/api/v3
TTN_LW_CONSOLE_UI_IS_BASE_URL=https://subdomain.example.com**:8885**/api/v3
TTN_LW_CONSOLE_UI_JS_BASE_URL=https://subdomain.example.com**:8885**/api/v3
TTN_LW_CONSOLE_UI_NS_BASE_URL=https://subdomain.example.com**:8885**/api/v3
TTN_LW_CONSOLE_UI_EDTC_BASE_URL=https://subdomain.example.com**:8885**/api/v3
TTN_LW_CONSOLE_UI_QRG_BASE_URL=https://subdomain.example.com**:8885**/api/v3
`

When we:

• stop apache (service apache2 stop)
• reconfigure ttn-stack to enable (80, 443 port - remove #)
• connect with web browser to main subdomain it gets the certificate sucessfully (on 80/443 ports)

`
#docker-compose.yml file
ports:
- '80:1885'
- '443:8885'
- '1882:1882'
- '8882:8882'
- '1883:1883'
- '8883:8883'
- '1884:1884'
- '8884:8884'

- '1885:1885'

- '8885:8885'

  - '8886:8886'
  - '1887:1887'
  - '8887:8887'
  - '1700:1700/udp'
env_file: '.env'

.env file standard ports used 443 - in this case ttn get letsencrypt certificate

TTN_LW_IS_EMAIL_NETWORK_CONSOLE_URL=https://subdomain.example.com/console
TTN_LW_IS_EMAIL_NETWORK_IDENTITY_SERVER_URL=https://subdomain.example.com/oauth

Intentionally omitting email provider config

TTN_LW_IS_OAUTH_UI_CANONICAL_URL=https://subdomain.example.com/oauth
TTN_LW_IS_OAUTH_UI_IS_BASE_URL=https://subdomain.example.com/api/v3

TTN_LW_CONSOLE_OAUTH_AUTHORIZE_URL=https://subdomain.example.com/oauth/authorize
TTN_LW_CONSOLE_OAUTH_TOKEN_URL=https://subdomain.example.com/oauth/token

TTN_LW_CONSOLE_UI_CANONICAL_URL=https://subdomain.example.com/console
TTN_LW_CONSOLE_UI_AS_BASE_URL=https://subdomain.example.com/api/v3
TTN_LW_CONSOLE_UI_GS_BASE_URL=https://subdomain.example.com/api/v3
TTN_LW_CONSOLE_UI_IS_BASE_URL=https://subdomain.example.com/api/v3
TTN_LW_CONSOLE_UI_JS_BASE_URL=https://subdomain.example.com/api/v3
TTN_LW_CONSOLE_UI_NS_BASE_URL=https://subdomain.example.com/api/v3
TTN_LW_CONSOLE_UI_EDTC_BASE_URL=https://subdomain.example.com/api/v3
TTN_LW_CONSOLE_UI_QRG_BASE_URL=https://subdomain.example.com/api/v3
`

Now if we have updated certificate we can switch back to first configuration (port 80/443 used by apache, ttn use 1885/8885 for console) - and its succefully open console on https://subdomain.domain.com:8885/

Is there a way to configure both ttn and apache on the same machine?
or force lets encrypt to get cerfication on different port?