Updated to 1.5.0 and now I have no connect bar

[steven86m]

Updated to 1.5.0 and now I have no connect bar agent's have been updated too!

[Mardok1]

same on 1.5.1 and disappears over time on working ones

Edit:

restart and it works for a while

after some time the domain https://mesh..... stops responding

[steven86m]

The buttons came back for now

[Mardok1]

thanks for the information, I'm still waiting patiently

[steven86m]

Still something wrong with it, if it's only lasting 2 hours

[wh1te909]

try this:

sudo systemctl disable --now meshcentral
sudo reboot
# wait for server to reboot and log back in
sudo -iu postgres psql -d meshcentral -c "CREATE INDEX IF NOT EXISTS ndxeventidsfkid ON eventids(fkid);" # do NOT ctrl +c this, might take a bit
sudo systemctl enable --now meshcentral

[Mardok1]

ndxeventidsfkid or ndx_eventids_fkid?

[ManPfe]

Root cause found: missing index on eventids.fkid makes MeshCentral 1.2.0 event-cleanup hang for ~99 min (blocks login → 504, web → 502)

This explains the disappearing connect bar / 502 / hanging Mesh login that several of us
have hit since TRMM moved to MeshCentral v1.2.0. It is a PostgreSQL schema issue with a
one-line fix.

Environment

• TRMM: v1.5.1
• MeshCentral: v1.2.0 (WAN mode, Production mode), PostgreSQL backend
• PostgreSQL: 15
• OS: Ubuntu 22.04.5 LTS, VPS, 23 GB RAM
• Node: v20.20.2

Symptoms

• TRMM connect bar disappears; Take Control / Remote Background return 502 Bad Gateway.
• Direct MeshCentral web login hangs and eventually returns 504 Gateway Time-out.
• TRMM agent still shows online (NATS unaffected).
• Recovers on restart, recurs after variable uptime — because the cleanup runs periodically,
  not on a fixed schedule.

Root cause

MeshCentral periodically runs DELETE FROM events WHERE time < $1 to trim the audit log.
The eventids table references events with ON DELETE CASCADE:

events.id <-- eventids.fkid (FK: fk_eventid, ON DELETE CASCADE)

events.time is indexed (ndxeventstime), so the SELECT side is fine. But there is no
index on the FK column eventids.fkid. PostgreSQL does not auto-create indexes on
referencing FK columns, so every cascaded delete has to seq-scan eventids. On a large
install this turns the cleanup into hundreds of thousands of full scans.

Caught live in pg_stat_activity — one cleanup query active for 1h39m:
pid 20995 | 01:39:21 | active | DELETE FROM events WHERE time < $1
While it runs it blocks other Mesh operations, which surfaces as 502 (no web) or 504 (hanging login).

Scale on the affected install

• events: 444,165 rows, 946 MB
• n_dead_tup = 0 (not bloat), events.time indexed — so it is specifically the unindexed FK.

Fix (one line)

CREATE INDEX IF NOT EXISTS ndx_eventids_fkid ON eventids (fkid);

Before / after (same delete, EXPLAIN ANALYZE)

• Before: cleanup query hung ~99 minutes (seq-scan of eventids per cascaded row).
• After: deleting 387,705 old rows finished in ~14 s total, with the cascade itself at
  Trigger for constraint fk_eventid: time=5850ms calls=387705. Normal incremental cleanup
  (only rows newly past retention) is then sub-second.

Suggested permanent fix

Add the index on eventids.fkid to MeshCentral's PostgreSQL schema init (Ylianst/MeshCentral),
so it survives upgrades and applies to all PG installs. Manually-added indexes get lost when
the schema is re-created on update.

Ruled out (to save time)

• Not OOM / RAM (23 GB total, ~20 GB free, swap 0, no OOM-killer entries)
• Not disk (root FS 19%)
• Not PostgreSQL down (PG healthy throughout, same PID, select 1 OK, max_connections=100,
  statement_timeout=0)
• Not Node version (/usr/bin/node = v20.20.2)
• Not orphaned processes (KillMode=control-group)
• Index on events.time exists; the Seq Scan on events in the plan is correct (the delete
  matches most rows) — the bottleneck is purely the unindexed FK cascade.

Happy to provide full \d events / \d eventids / EXPLAIN ANALYZE output.

[Mardok1]

something like that?

[ManPfe]

Yes, exactly that — connecting to the meshcentral DB first (your \c meshcentral) and
then CREATE INDEX IF NOT EXISTS ndx_eventids_fkid ON eventids (fkid); is the right fix.
CREATE INDEX confirms it's in place.

To verify it actually helps on your install, you can time the cleanup once:
```sql
EXPLAIN ANALYZE DELETE FROM events WHERE time < now() - interval '90 days';
```
Look at the bottom line Trigger for constraint fk_eventid: ... and Execution Time —
it should now be seconds instead of running for many minutes. (Wrap it in
BEGIN; ... ROLLBACK; first if you don't want to delete anything yet.)

Two notes:

• This also clears out old audit events, so afterwards run VACUUM (ANALYZE) events; to
  tidy up.
• The index is not upgrade-safe — a future MeshCentral schema re-init may drop it. It
  really needs to land in the upstream Ylianst/MeshCentral schema, which is why getting this
  into the schema (not just our manual workaround) matters.

[Mardok1]

Everything has been working for over 8 hours without any problems, thank you very much for your help

[Scriptkiddi]

Worked for us as well index creation took hours but after that its working fine