Automate adding servers to master server?

[grephappiness]

I can see it's possible to automate deploying cronicle onto additional worker nodes. You would run the install command, copy over the master config then start the service.

The part I'm not clear on is how I could automate adding the worker nodes to the cronicle master server. Right now It looks like the only option is to add them via the web UI? I checked through the API calls and didn't see any option for add_server.

Thank you for your time.

[jhuckaby]

Hello! I do apologize -- the API you want exists of course, but it's not documented, and it's an "admin-only" API. To make matters worse, due to reasons I can't remember now, Cronicle API Keys cannot have the "admin" privilege. I honestly don't remember why I put in this restriction -- possibly due to security reasons.

Anyway, you should be able to do it by following these steps. Warning: This is a bit of a hack.

First, create the API key you will be using to add servers in the UI. Second, shut down the Cronicle service on your master server:

/opt/cronicle/bin/control.sh stop

Then execute these commands as root:

cd /opt/cronicle
bin/storage-cli.js edit global/api_keys/0

This command should spawn the editor of your preference in the terminal (e.g. vi, whatever is in the EDITOR environment variable), and load up the API keys JSON record from the DB, pretty-printed for your convenience. You can edit this record in the terminal, then "save & exit" your editor, and the script will submit the record back into the Cronicle DB. Please be very careful about JSON syntax here. It's quite picky.

Locate the API key you want to "admin-ify" in the list, and then locate the privileges object within. Inside that object, add an admin key and set it to the number 1. Here is an example:

{
	"privileges": {
		"admin": 1
	},
	"active": "1",
	"title": "Joe's Test Key",
	"description": "It does a few things.",
	"id": "93fe0f04",
	"key": "0095f5b664b93304d5f8b1a61df605fb",
	"username": "admin",
	"modified": 1454300467,
	"created": 1441659908
}

The privileges object shouldn't contain any other keys (delete them if they exist). It should only contain the admin key by itself. Here is a closer look:

"privileges": {
	"admin": 1
},

Once you make this change to your API key, save the file and exit. The storage CLI script should detect that your editor exited and saved the temp file, and it should re-submit it back into the Cronicle DB. Please look closely at the console output for any errors here (JSON syntax errors, storage write errors, etc.).

Now you are ready to use the API key, which has been effectively promoted to administrator, against the application's wishes. I am honestly not sure if this will cause problems, but I think it should work. Just be super careful with it.

Remember to restart the Cronicle service on the master server:

/opt/cronicle/bin/control.sh start

Here is the undocumented API you want to hit:

http://MASTER_SERVER:PORT/api/app/add_server

This API follows the same rules as the others -- see the Cronicle JSON REST API docs for details. It expects two parameters, which you can either add as query string params on the URL and do an HTTP GET, or JSON keys with an HTTP POST:

Parameter Name	Description
hostname	Hostname or IP address of the server you want to add to the Cronicle cluster.
api_key	The API Key you want to use for authentication.

Example URL (HTTP GET):

http://cronicle.mycompany.com:3012/api/app/add_server?hostname=worker01.mycompany.com&api_key=0095f5b664b93304d5f8b1a61df605fb

Make sure you check the response back from this API for any errors. See the Cronicle Standard Response Format in the docs. Check the code response JSON key.

Remember, the server has to be "reachable" from the master server. The hostname needs to be resolvable in DNS, and the IP needs to be routable from the master server, and visa-versa.

Good luck!

-- Joe

[jhuckaby]

Similarly, here is how to remove a server using the undocumented APIs:

http://MASTER_SERVER:PORT/api/app/remove_server

This API follows the same rules as the others -- see the Cronicle JSON REST API docs for details. It expects two parameters, which you can either add as query string params on the URL and do an HTTP GET, or JSON keys with an HTTP POST:

Parameter Name	Description
hostname	Hostname or IP address of the server you want to remove from the Cronicle cluster.
api_key	The API Key you want to use for authentication.

Example URL (HTTP GET):

http://cronicle.mycompany.com:3012/api/app/remove_server?hostname=worker01.mycompany.com&api_key=0095f5b664b93304d5f8b1a61df605fb

Make sure you check the response back from this API for any errors. See the Cronicle Standard Response Format in the docs. Check the code response JSON key.

[danwillo]

Hi, does this still work in v0.9.12 ?

I've followed the steps but alas am getting the following response from the API:

{"code":"api","description":"API Key cannot use administrator features"}

API Key hack:

"type": "list_page",
"items": [
{
"privileges": {
"admin": 1
},
"key": "e4e77328d6a6e751b506272935c69d31",
"active": "1",
"title": "admin",
"description": "testing",
"id": "kl9cjas3f01",
"username": "admin",
"modified": 1665996821,
"created": 1665996496
}
]
}

API call

curl -v 'http://localhost:3012/api/app/add_server' -d hostname=172.17.0.3 -d api_key=e4e77328d6a6e751b506272935c69d31

any help appreciated!

thanks

[jhuckaby]

@danwillo Hey Dan, so sorry, this is totally a hack. I haven't had time to actually test it. These instructions here in this discussion are very much "at your own risk". I have no idea if they'll work. I just vaguely remember this NOT working back in 2014 when I first wrote Cronicle.

I searched for that error message and found a strange "block" where I deliberately prevent API keys from ever becoming admins. I have no memory of adding this code, as it was 8 years ago now. I swear I had a good reason for adding it, but it eludes me. I've just now removed the block with this commit:

2085e2c

And pushed a new release: https://github.com/jhuckaby/Cronicle/releases/tag/v0.9.13

Again, I have to stress -- I have no idea if this will actually work. I've never tested any API key administrator stuff. This is all theoretical.

Good luck.

[danwillo]

Hi Joe, thanks for that ! I shall give it a whirl today and let you know how it goes. completely understand the caveats around using the admin role on api keys, and tbh its not a complete blocker for us really. just would be nice not have to add manually.

side question - are you actively working on Orchestra ? I wondered how that was going?

cheers

Dan

[jhuckaby]

@danwillo Yup, I'm still plugging away at Orchestra! It's taking me much longer than I'd originally hoped, as life keeps getting in the way (as it does 😊), but it's most definitely still happening.