Service Unavailable with no obvious errors

[bencullpinch]

Hi Team,

The server stopped responding to requests overnight and now all I get is service unavailable.

I've tried:

• Restarting the server
• Restarting the service
• Running the seq admin application to reconfigure the instance

I've tried running the service in interactive mode and get the following output:

[22:12:28 INF] Seq 2023.4.10219 running on OS Microsoft Windows Server 2022 Datacenter Azure Edition
[22:12:29 INF] Configuring HTTP.sys web server
[22:12:29 INF] Opening event store at F:\SeqData\Stream\stream.flare
[22:12:29 INF] Ingestion enabled
[22:12:29 INF] Opening metastore F:\SeqData\Documents\metastore.flare
[22:12:30 INF] Storage subsystem available
[22:12:30 INF] Seq listening on ["redacted"]
[22:12:30 INF] Shadow-copy app host executable completed in 229.2 ms
[22:12:39 INF] Lock shadow-copied app host executable and its dependencies completed in 9470.7 ms
[22:12:40 INF] Started seqcli.exe process 5436 for appinstance-1090
[22:17:40 INF] Metrics sampled

Service Unavailable is received both from the public URL and internally on the VM itself visting https://localhost

Note, disks are not full.

[bencullpinch]

A suspicious coindidence is an updated lets encrypt certificate at around the time of the requests stopping. It appears to be served correctly though.

[nblumhardt]

(Reaching out via the Slack we're both on.)

[bencullpinch]

OK, so I've found the issue. For us, it was the domain binding. When using the actual domain in the Listen URI field (eg. https://example.com) the server would show Service Unavailable 503. Using localhost instead (https://localhost) serves the app correctly. The only caveat is that external URLs that are shown in the app show as localhost, so it's difficult to share links, and alerts show the wrong url.

The reason the lets encrypt was suspicious, and actually caused this issue, is because when a cert is renewed we run a powershell script to rebind the listen uri with the new thumpprint. This script was using the full domain, and thus the problem began.

I'll leave the discussion open in cas ethere's a solution to the domain binding problem, but the urgency is now low :)

[nblumhardt]

Thanks for the follow-up, Ben! That's strange - unless there's some kind of load balancer/reverse proxy/app gateway in front of the VM?

RE inbound links generated in Seq, you can fix this with:

seq config set -k api.canonicalUri -v https://example.com
seq service restart

The "canonical URI" setting only affects link generation, and won't have any impact on how the API is served.

Hope this helps!

[bencullpinch]

There's nothing in front of the VM, unless Azure is doing something funky I don't know about. Either way this canonical uri has done the job perfectly and I'm back running 100%.

Thank you for the swift response and your time today 😊 You're a legend 😎