Fly.io deployment with mounts does not work anymore

[ribtoks]

Hi

I used to deploy picoshare to Fly.io with the litestream option successfully before. However, this time I tried the one with the mounted volume and it does not work anymore: I cannot open the webpage and fly.io logs have " could not find a good candidate within 90 attempts at load balancing"

version: 1.4.1 (also I tried 1.3.8)

Full logs

 2024-01-02T15:59:26.341 app[148e5996b5d598] arn [info] INFO Starting init (commit: 15238e9)...
2024-01-02T15:59:26.366 app[148e5996b5d598] arn [info] INFO Mounting /dev/vdb at /data w/ uid: 0, gid: 0 and chmod 0755
2024-01-02T15:59:26.368 app[148e5996b5d598] arn [info] INFO Resized /data to 1056964608 bytes
2024-01-02T15:59:26.369 app[148e5996b5d598] arn [info] INFO Preparing to run: `/app/docker-entrypoint -db /data/store.db` as root
2024-01-02T15:59:26.374 app[148e5996b5d598] arn [info] INFO [fly api proxy] listening at /.fly/api
2024-01-02T15:59:26.378 app[148e5996b5d598] arn [info] 2024/01/02 15:59:26 listening on [fdaa:1:afca:a7b:1e9:59f1:6467:2]:22 (DNS: [fdaa::3]:53)
2024-01-02T15:59:26.390 app[148e5996b5d598] arn [info] Starting without litestream
2024-01-02T15:59:26.390 app[148e5996b5d598] arn [info] ++ echo false
2024-01-02T15:59:26.390 app[148e5996b5d598] arn [info] + PS_LAUNCH_CMD='/app/picoshare -db /data/store.db'
2024-01-02T15:59:26.390 app[148e5996b5d598] arn [info] + [[ false == \t\r\u\e ]]
2024-01-02T15:59:26.390 app[148e5996b5d598] arn [info] + echo 'Starting without litestream'
2024-01-02T15:59:26.390 app[148e5996b5d598] arn [info] + eval 'exec /app/picoshare -db /data/store.db'
2024-01-02T15:59:26.390 app[148e5996b5d598] arn [info] ++ exec /app/picoshare -db /data/store.db
2024-01-02T15:59:26.414 app[148e5996b5d598] arn [info] 2024/01/02 15:59:26 initializing random seed
2024-01-02T15:59:26.416 app[148e5996b5d598] arn [info] 2024/01/02 15:59:26 main.go:26: starting picoshare server
2024-01-02T15:59:26.416 app[148e5996b5d598] arn [info] 2024/01/02 15:59:26 sqlite.go:38: reading DB from /data/store.db
2024-01-02T15:59:26.419 app[148e5996b5d598] arn [info] 2024/01/02 15:59:26 migrations.go:33: migration counter: 7/7
2024-01-02T15:59:26.420 app[148e5996b5d598] arn [info] 2024/01/02 15:59:26 main.go:64: listening on 8080
2024-01-02T16:01:13.113 proxy[148e5996b5d598] ams [error] could not find a good candidate within 90 attempts at load balancing 

fly.toml

app = "picoshare-jh25tc"
primary_region = "arn"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]
  PORT = "8080"
  PS_BEHIND_PROXY="yes"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

[mounts]
  source="pico_data"
  destination="/data"

[mtlynch]

Is it possible it's a Fly outage and not a PicoShare issue? In the logs, PicoShare is serving successfully:

2024-01-02T15:59:26.419 app[148e5996b5d598] arn [info] 2024/01/02 15:59:26 migrations.go:33: migration counter: 7/7
2024-01-02T15:59:26.420 app[148e5996b5d598] arn [info] 2024/01/02 15:59:26 main.go:64: listening on 8080

Is the config you're using intended to sync with Litestream? It's currently creating its own local database and not syncing to Litestream:

2024-01-02T15:59:26.390 app[148e5996b5d598] arn [info] + echo 'Starting without litestream'

If you're troubleshooting this, I recommend either turning off the volume or turning off Litestream entirely to prevent a newly-created DB from overwriting the replicated snapshots through Litestream. When you've got it working again, use a fresh Fly volume so that the DB on your pico_data volume isn't going to overwrite the replicated data on Litestream.

[ribtoks]

Yeah, I saw that picoshare has started listening, but I assumed the problem was with healthchecks, since fly.io "couldn't find a healthy load balancer". However, I redeployed today again and it seems to work, so might be a transient fly.io problem