Server is not eligible for master after upgrade

[julian-weinert]

Summary

After not having used Cronicle for quite some time I fired it up again today and decided to update everything.
I installed software updates, OS patches and the current LTS version of Node.js. After checking Cronicle I found an available update from 0.8.-something (maybe 4?).

After not finding update instructions I ran the installer again which detected the installation without issues and showed me it does an update.

Having the update completed I wanted to login but got stuck with waiting for master. I waited for a long time and then checked the logs. Cronicle says Server is not eligible for master -- it will be a slave only. I dug around a bit and found the troubleshooting guide in the wiki and worked through that.

Running storage-cli.js get global/server_groups/0 revealed that there is actually no master group defined.

{"type": "list_page",
"items": [
	{
		"id": "allgrp",
		"title": "All Servers",
		"regexp": ".+",
		"master": 0
	}
]}

The server also never changed hostname or IP address, also confirmed using the Node code mentioned in the guide.

Did I simply miss a version step that introduced groups and so I ended up without one? How would I go about adding a group manually?

Any help is appreciated!

Operating system and version?

Debian 10.11 (Buster)

Node.js version?

v16.13.2

Cronicle software version?

v0.9.2

Are you using a multi-server setup, or just a single server?

Single server

Are you using the filesystem as back-end storage, or S3/Couchbase?

Filesystem

Log Excerpts

[1643667283.224][2022-01-31 23:14:43][cronicle][16294][Storage][transaction][get][global/servers/0][{"elapsed_ms":1.517}]
[1643667283.225][2022-01-31 23:14:43][cronicle][16294][Storage][debug][9][Removing reader from shared lock: |global/servers][{"readers":0}]
[1643667283.225][2022-01-31 23:14:43][cronicle][16294][Storage][debug][9][Unlocking key: |global/servers (0 clients waiting)][]
[1643667283.226][2022-01-31 23:14:43][cronicle][16294][Storage][debug][9][Fetching 0 items at position 0 from list: global/server_groups][]
[1643667283.226][2022-01-31 23:14:43][cronicle][16294][Storage][debug][9][Requesting shared lock: |global/server_groups][]
[1643667283.226][2022-01-31 23:14:43][cronicle][16294][Storage][debug][9][Locked key in shared mode: |global/server_groups][]
[1643667283.227][2022-01-31 23:14:43][cronicle][16294][Storage][debug][9][Loading list: global/server_groups][]

==> logs/Filesystem.log <==
[1643667283.227][2022-01-31 23:14:43][cronicle][16294][Filesystem][debug][9][Fetching Object: global/server_groups][data/global/bf/10/4a/bf104a19df63f5227cc0e457c353afa7.json]
[1643667283.228][2022-01-31 23:14:43][cronicle][16294][Filesystem][debug][9][JSON fetch complete: global/server_groups][{"page_size":50,"first_page":0,"last_page":0,"length":1,"type":"list"}]

==> logs/Filesystem.log <==
[1643667283.227][2022-01-31 23:14:43][cronicle][16294][Filesystem][debug][9][Fetching Object: global/server_groups][data/global/bf/10/4a/bf104a19df63f5227cc0e457c353afa7.json]
[1643667283.228][2022-01-31 23:14:43][cronicle][16294][Filesystem][debug][9][JSON fetch complete: global/server_groups][{"page_size":50,"first_page":0,"last_page":0,"length":1,"type":"list"}]

==> logs/Storage.log <==
[1643667283.228][2022-01-31 23:14:43][cronicle][16294][Storage][transaction][get][global/server_groups][{"elapsed_ms":1.082}]
[1643667283.228][2022-01-31 23:14:43][cronicle][16294][Storage][debug][9][Loading list page: global/server_groups/0][]

==> logs/Filesystem.log <==
[1643667283.229][2022-01-31 23:14:43][cronicle][16294][Filesystem][debug][9][Fetching Object: global/server_groups/0][data/global/70/a4/5b/70a45b37adf20d85be16525c9a8e8e03.json]
[1643667283.229][2022-01-31 23:14:43][cronicle][16294][Filesystem][debug][9][JSON fetch complete: global/server_groups/0][{"type":"list_page","items":[{"id":"allgrp","title":"All Servers","regexp":".+","master":0}]}]

==> logs/Cronicle.log <==
[1643667283.231][2022-01-31 23:14:43][cronicle][16294][Cronicle][debug][4][Server is not eligible for master -- it will be a slave only][]
[1643667283.231][2022-01-31 23:14:43][cronicle][16294][Cronicle][debug][2][Startup complete, entering main loop][]

[jhuckaby]

What a bizarre problem. I've never heard of this one before. I can assure you that Cronicle never deletes any data. There is no code written anywhere that would go in and surgically remove your master group. That is just utterly bizarre.

I'm curious if you grep over your log archives and try to find anything matching [group_delete]. You might be able to see when it happened, and which user did it.

Anyway, we need to get the master group re-added, so you'll have to edit two JSON records manually. The first one is the record you included above, but use edit instead of get to edit it (will spawn vi or whatever your Linux editor of choice is from the EDITOR environment variable).

First, make sure Cronicle is not running for this procedure:

/opt/cronicle/bin/control.sh stop

Next, edit this record:

/opt/cronicle/bin/storage-cli.js edit global/server_groups/0

Add your master group like this. Please pay special attention to the regexp property, and make sure this will match your own master server hostname. It looks like your server hostname may simply be cronicle judging by your log excerpts above, but you may have altered those for security purposes.

{
	"type": "list_page",
	"items": [
		{
			"id": "allgrp",
			"title": "All Servers",
			"regexp": ".+",
			"master": 0
		},
		{
			"id": "mastergrp",
			"title": "Master Servers",
			"regexp": "cronicle",
			"master": 1
		}
	]
}

Once you save and exit your editor (vi or other) the CLI will detect the temp file change and actually commit the data back to Cronicle storage.

Next, please edit this record (same path, but without the /0):

/opt/cronicle/bin/storage-cli.js edit global/server_groups

This is the list header. Make sure the length property is 2, which reflects the actual number of items in the list record:

{
	"page_size": 50,
	"first_page": 0,
	"last_page": 0,
	"length": 2,
	"type": "list"
}

Again, save and exit your editor (vi or other) the CLI will detect the temp file change and actually commit the data back to Cronicle storage.

Now try starting Cronicle, waiting for one full minute, and see if your server becomes master.

/opt/cronicle/bin/control.sh start

If this process doesn't work, you can try doing a data export, then editing the export file (it's just plain text), make the necessary changes to the list records, then doing an import. Instructions here: https://github.com/jhuckaby/Cronicle#data-import-and-export

[julian-weinert]

Yay! That did the trick!
After going through your troubleshooting guide I actually tried adding the group, but didn't go the route of trying to edit groups length entry, since I wasn't sure how things go together.

The thing is that I only installed cronicle, tested a few things and then the project was on hold for some time, so I didn't use it for a long time. I also couldn't find a delete event in the log archives.
So this stays a mystery I guess...

Thanks for your quick help :)