Docker compose: Unable to reach seq running behind traefik

[johanndev]

Hi!

I apologize in advance if this question is out of scope for this discussion forum, but I'm hoping that someone here has expertise with both traefik + seq and is able to point me in the right direction :)

I'm currently trying to set up a small ASP.NET Core web application using docker compose. I want to use Seq to capture the log output from the various services, unfortunately I'm unable to reach the Seq UI. I'm suspecting that maybe certain env variables (SEQ_API_LISTENURIS? SEQ_API_CANONICALURI?) have to be set for my setup to work, but I haven't been able to find the right combination.

EDIT: I tried to set: SEQ_API_CANONICALURI: "https://staging.example.com/seq", but it didn't make a difference.
EDIT 2: I realized that I should add a sample that is runnable without setting up a custom domain - this compose file runs everything on localhost and exhibits the same behavior as the sample below.

docker-compose.yml for localhost

services:
  traefik:
    image: traefik:2.6
    container_name: traefik
    restart: unless-stopped
    command:
      - "--providers.docker=true"
      - "--api.dashboard=true"
      - "--log.level=DEBUG"
      - "--api.debug=true"
      - "--entrypoints.web.address=:80"
    ports:
      - 80:80
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=web"
      - "traefik.http.routers.traefik.service=api@internal"
      - 'traefik.http.routers.traefik.middlewares=strip'
      - 'traefik.http.middlewares.strip.stripprefix.prefixes=/traefik'
      # Dashboard
      - "traefik.http.routers.dashboard.rule=Host(`localhost`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=auth"

  whoami:
    image: traefik/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`localhost`) && PathPrefix(`/whoami`)"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.http.services.whoami.loadbalancer.server.port=80"
    
  seq:
    image: datalust/seq:latest
    restart: unless-stopped 
    environment:
      ACCEPT_EULA: "Y"
      SEQ_DIAGNOSTICS_INTERNALLOGGINGLEVEL: "Debug"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.seq.rule=Host(`localhost`) && PathPrefix(`/seq`)"
      - "traefik.http.routers.seq.entrypoints=web"
      - "traefik.http.services.seq.loadbalancer.server.port=80"
    volumes:
      - ./seq_data:/data
    ports:
      - "5341:5341"
      - "8084:80"

docker-compose.yml

services:
  traefik:
    image: traefik:2.6
    container_name: traefik
    restart: unless-stopped
    command:
      # Tell Traefik to discover containers using the Docker API
      - --providers.docker=true
      # Enable the Trafik dashboard
      - --api.dashboard=true
      # Log to stdout
      - "--log.level=DEBUG"
      - "--api.debug=true"
      # Set up LetsEncrypt
      - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
      - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.letsencrypt.acme.email=<snip>
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
      # Set up an insecure listener that redirects all traffic to TLS
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      # Set up the TLS configuration for our websecure listener
      - --entrypoints.websecure.http.tls=true
      - --entrypoints.websecure.http.tls.certResolver=letsencrypt
      - --entrypoints.websecure.http.tls.domains[0].main=staging.example.com
      - --entrypoints.websecure.http.tls.domains[0].sans=*.staging.example.com
    environment:
      - CLOUDFLARE_EMAIL=<snip>
      - CLOUDFLARE_DNS_API_TOKEN=<snip>
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./letsencrypt:/letsencrypt
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik.service=api@internal"
      - 'traefik.http.routers.traefik.middlewares=strip'
      - 'traefik.http.middlewares.strip.stripprefix.prefixes=/traefik'
      # Dashboard
      - "traefik.http.routers.dashboard.rule=Host(`staging.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=<snip>"

  whoami:
    image: traefik/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`staging.example.com`) && PathPrefix(`/whoami`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=letsencrypt"
      - "traefik.http.routers.whoami.tls=true"
	  # This line shows how to explictly set the port that traefik forwards requests to
      - "traefik.http.services.whoami.loadbalancer.server.port=80"
    
  seq:
    image: datalust/seq:latest
    restart: unless-stopped 
    environment:
      ACCEPT_EULA: "Y"
      SEQ_DIAGNOSTICS_INTERNALLOGGINGLEVEL: "Debug"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.seq.rule=Host(`staging.example.com`) && PathPrefix(`/seq`)"
      - "traefik.http.routers.seq.entrypoints=websecure"
      - "traefik.http.routers.seq.tls.certresolver=letsencrypt"
      - "traefik.http.routers.seq.tls=true"
      - "traefik.http.services.seq.loadbalancer.server.port=80"
    volumes:
      - ./seq_data:/data
    ports:
      - "5341:5341"
      - "8084:80"

As you can see, I also included an instance of the traefik/whoami for testing purposes. When I navigate to staging.example.com/whoami, everything works as expected: The request is routed to the whoami service and the result can be viewed in the browser.

When I navigate to staging.example.com/seq, I only get a blank page - inspecting the request in the browser devtools shows that the server responded with a 404.

Opening a shell on the host and running: curl localhost:8084 responds with the Seq UI html page, indicating that the service is in fact running; the log entries for seq also seem to indicate that the service was started successfully

docker compose logs seq:

deploy-seq-1  | ────────────────────────────────────────
deploy-seq-1  |   Seq ♦ Machine data, for humans.
deploy-seq-1  | ─────────── © 2021 Datalust Pty Ltd ────
deploy-seq-1  |
deploy-seq-1  | Running as server; press Ctrl+C to exit.
deploy-seq-1  |
deploy-seq-1  | [13:49:21 INF] Seq "2021.4.6986" running on OS "Linux 5.4.0-94-generic #106-Ubuntu SMP <snip>"
deploy-seq-1  | [13:49:22 INF] Seq detected 2029.088768 MB of RAM
deploy-seq-1  | [13:49:22 INF] Opening event store at "/data/Stream/stream.flare"
deploy-seq-1  | [13:49:22 INF] Ingestion enabled
deploy-seq-1  | [13:49:22 INF] Cache warm-up is required
deploy-seq-1  | [13:49:22 INF] Opening metastore "/data/Documents/documents.lmdb"
deploy-seq-1  | [13:49:22 INF] Initial memory cache warm-up "completed" in 29.6351 ms
deploy-seq-1  | [13:49:22 INF] 19 segments warmed up ({"Schemata": 14, "Strings": 32})
deploy-seq-1  | [13:49:22 INF] Storage subsystem available
deploy-seq-1  | [13:49:22 INF] Seq listening on ["http://localhost/", "https://localhost/", "http://localhost:5341/", "https://localhost:45341/"]

Running docker compose logs traefik and navigating to staging.example.com/seq yields:

traefik       | time="<snip>" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" <See request below *>
traefik       | time="<snip>" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" <See request below *>  ForwardURL="http://192.168.0.2:80"
traefik       | time="<snip>" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" <See request below *>

• Request body:

{
	"Method": "GET",
	"URL": {
		"Scheme": "",
		"Opaque": "",
		"User": null,
		"Host": "",
		"Path": "/seq",
		"RawPath": "",
		"ForceQuery": false,
		"RawQuery": "",
		"Fragment": "",
		"RawFragment": ""
	},
	"Proto": "HTTP/2.0",
	"ProtoMajor": 2,
	"ProtoMinor": 0,
	"Header": {
		"Accept": [
			"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"
		],
		"Accept-Encoding": [
			"gzip"
		],
		"Accept-Language": [
			"en,de-DE;q=0.5"
		],
		"Cache-Control": [
			"max-age=0"
		],
		"Cdn-Loop": [
			"cloudflare"
		],
		"Cf-Connecting-Ip": [
			"<snip>1"
		],
		"Cf-Ipcountry": [
			"<snip>"
		],
		"Cf-Ray": [
			"<snip>"
		],
		"Cookie": [
			"<snip>"
		],
		"Dnt": [
			"1"
		],
		"Sec-Fetch-Dest": [
			"document"
		],
		"Sec-Fetch-Mode": [
			"navigate"
		],
		"Sec-Fetch-Site": [
			"none"
		],
		"Sec-Fetch-User": [
			"?1"
		],
		"Upgrade-Insecure-Requests": [
			"1"
		],
		"User-Agent": [
			"<snip>"
		],
		"X-Forwarded-Host": [
			"staging.example.com"
		],
		"X-Forwarded-Port": [
			"443"
		],
		"X-Forwarded-Proto": [
			"https"
		],
		"X-Forwarded-Server": [
			"<snip>"
		],
		"X-Real-Ip": [
			"<snip>"
		]
	},
	"ContentLength": 0,
	"TransferEncoding": null,
	"Host": "staging.example.com",
	"Form": null,
	"PostForm": null,
	"MultipartForm": null,
	"Trailer": null,
	"RemoteAddr": "<snip>:23362",
	"RequestURI": "/seq",
	"TLS": null
}

Does anybody have an idea what I could try next?
Thanks!

[nblumhardt]

Hi!

I've managed to get as far as having traefik serve Seq in your compose file by adding the strip-prefix middleware, and SEQ_API_CANONICALURI.

  seq:
    image: datalust/seq:latest
    restart: unless-stopped 
    environment:
      ACCEPT_EULA: "Y"
      SEQ_DIAGNOSTICS_INTERNALLOGGINGLEVEL: "Debug"
      SEQ_API_CANONICALURI: "http://localhost/seq"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.seq.rule=Host(`localhost`) && PathPrefix(`/seq`)"
      - "traefik.http.middlewares.autodetect.contenttype.autodetect=false"
      - "traefik.http.middlewares.strip-seq.stripprefix.prefixes=/seq"
      - "traefik.http.routers.seq.entrypoints=web"
      - "traefik.http.routers.seq.middlewares=strip-seq,autodetect"
      - "traefik.http.services.seq.loadbalancer.server.port=80"
    volumes:
      - ./seq_data:/data
    ports:
      - "5341:5341"
      - "8084:80"

However, I'm unable to figure out how to turn off traefik's broken content-type autodetection; you can see my attempts above, based on https://doc.traefik.io/traefik/middlewares/http/contenttype/ - probably just my lack of traefik knowledge hitting me, here.

The result is that the UI is served, but subsequent requests fail due to mismatched content types.

Would be great if you can spot what's going wrong - crossing my fingers :-)

[johanndev]

Progress :)

May I ask why you suspect the content-type detection to be the culprit here? If I remove the autodetect middleware from the configuration, the service seems to behave exactly the same.

(I discovered the access log flag for traefik, which makes debuging a little easier: Just add - "--accesslog=true" to traefiks command array :))

Access log for http://localhost/seq:

traefik               | 172.20.0.1 - - [20/Jan/2022:07:54:17 +0000] "GET /seq HTTP/1.1" 200 2487 "-" "-" 18 "seq@docker" "http://172.20.0.3:80" 2ms
traefik               | 172.20.0.1 - - [20/Jan/2022:07:54:17 +0000] "GET /seq-ui2-1639983282608.min.css HTTP/1.1" 404 0 "-" "-" 19 "seq@docker" "http://172.20.0.3:80" 1ms
traefik               | 172.20.0.1 - - [20/Jan/2022:07:54:17 +0000] "GET /vendor-1639983282608.min.js HTTP/1.1" 404 19 "-" "-" 20 "-" "-" 0ms
traefik               | 172.20.0.1 - - [20/Jan/2022:07:54:17 +0000] "GET /theme/styles.css HTTP/1.1" 404 19 "-" "-" 21 "-" "-" 0ms
traefik               | 172.20.0.1 - - [20/Jan/2022:07:54:17 +0000] "GET /seq-ui2-1639983282608.min.js HTTP/1.1" 404 0 "-" "-" 22 "seq@docker" "http://172.20.0.3:80" 1ms
traefik               | 172.20.0.1 - - [20/Jan/2022:07:54:17 +0000] "GET /seq-ui2-tpl-1639983282608.min.js HTTP/1.1" 404 0 "-" "-" 23 "seq@docker" "http://172.20.0.3:80" 1ms
traefik               | 172.20.0.1 - - [20/Jan/2022:07:54:17 +0000] "GET /seq-ui2-tpl-1639983282608.min.js HTTP/1.1" 404 0 "-" "-" 24 "seq@docker" "http://172.20.0.3:80" 1ms
traefik               | 172.20.0.1 - - [20/Jan/2022:07:54:17 +0000] "GET /img/favicon.png HTTP/1.1" 404 19 "-" "-" 25 "-" "-" 0ms

The first request (for the initial route) now succeeds, all other requests fail with a 404 - I'm not sure if the problems originate from the strip prefix middleware, as a direct request to e.g. http://localhost/seq/vendor-1639983282608.min.js works.

One possible solution would be to inject the value of SEQ_API_CANONICALURI into the index page:

<!doctype html>
<html lang="en" ng-app="seq-ui2" ng-strict-di>
<head>
  <!-- 👇  Inject SEQ_API_CANONICALURI here 👇  -->
  <base href="http://localhost/seq/">
...

I hot patched a running container on my machine and it now works when accessing the instance via http://localhost/seq, but breaks when accessed from localhost:8084 (which would be used during dev time). This could be solved with only setting the env variable on the hosted system...

What do you think @nblumhardt?

[johanndev]

Setting the base href to <base href="/seq/"> also works. As stated in https://doc.traefik.io/traefik/middlewares/http/stripprefix/ (and confirmed in my experiments), this path segment is provided by traefik via the X-Forwarded-Prefix header to the backend service. So instead of injecting the SEQ_API_CANONICALURI variable, the value of this header (+ a trailing /) could be used in the html head.

[nblumhardt]

Ah, thanks for the further investigation, @johanndev!

I was able to get everything working with this compose snippet, where the Seq canonical URI and path prefix carry the trailing slash:

  seq:
    image: datalust/seq:latest
    restart: unless-stopped 
    environment:
      ACCEPT_EULA: "Y"
      SEQ_DIAGNOSTICS_INTERNALLOGGINGLEVEL: "Debug"
      SEQ_API_CANONICALURI: "http://localhost/seq/"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.seq.rule=Host(`localhost`) && PathPrefix(`/seq/`)"
      - "traefik.http.middlewares.autodetect.contenttype.autodetect=false"
      - "traefik.http.middlewares.strip-seq.stripprefix.prefixes=/seq"
      - "traefik.http.routers.seq.entrypoints=web"
      - "traefik.http.routers.seq.middlewares=strip-seq,autodetect"
      - "traefik.http.services.seq.loadbalancer.server.port=80"
    volumes:
      - ./seq_data:/data
    ports:
      - "5341:5341"
      - "8084:80"

An additional redirect from /seq to /seq/ would ensure users can reach Seq at the shorter/more obvious URL.

In non-proxied configurations, if it's listening under a specific path, Seq checks for the trailing / and adds it if necessary. I think we have an opportunity to make this use case simpler and cleaner, along the lines you've suggested. We'll give it some further thought!

Best regards,
Nick

[johanndev]

Thank you very much!

Best regards,
Johann