Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ansible: Install AVA as a systemd service #151

Merged
merged 25 commits into from
Jul 14, 2020

Conversation

moreati
Copy link
Contributor

@moreati moreati commented May 9, 2020

This playbook

  • Installs Gecko dependencies (Ubuntu 20.04 only for now)
  • Clones & builds ava-labs/gecko
  • Creates an ava user
  • Installs Gecko in /usr/bin
  • Creates and installs a staking certificate
  • Installs Gecko as a systemd service called "ava"
  • Configures /var/lib/ava/db as the database
  • Configures /var/log/ava as the log destination
  • Starts the service
An example run (click to expand)
$ ansible-playbook -i inventory.yml service_playbook.yml

PLAY [Configure AVA service] ********************************************************************************

TASK [Gathering Facts] **************************************************************************************
ok: [droplet]

TASK [ava-base : Install deps] ******************************************************************************
ok: [droplet]

TASK [gopath : Set GOPATH] **********************************************************************************
ok: [droplet]

TASK [ava-build : Update git clone] *************************************************************************
ok: [droplet]

TASK [ava-build : Build project] ****************************************************************************
changed: [droplet]

TASK [ava-certs : Create staker key] ************************************************************************
ok: [droplet]

TASK [ava-certs : Create staker certificate request] ********************************************************
ok: [droplet]

TASK [ava-certs : Create staker certificate] ****************************************************************
ok: [droplet]

TASK [ava-user : Create AVA daemon group] *******************************************************************
ok: [droplet]

TASK [ava-user : Create AVA daemon user] ********************************************************************
[WARNING]: The value False (type bool) in a string field was converted to 'False' (type string). If this
does not look like what you expect, quote the entire value to ensure it does not change.
ok: [droplet]

TASK [ava-install : Create directories] *********************************************************************
ok: [droplet] => (item=/var/lib/ava)
ok: [droplet] => (item=/var/lib/ava/keys)
ok: [droplet] => (item=/var/log/ava)
ok: [droplet] => (item=/usr/lib/ava/plugins)

TASK [ava-install : Install binary] *************************************************************************
ok: [droplet]

TASK [ava-install : Install plugins] ************************************************************************
ok: [droplet] => (item={'path': '~/go/src/github.com/ava-labs/gecko/build/plugins/evm'})

TASK [ava-install : Install staking files] ******************************************************************
ok: [droplet] => (item={'src': '~/go/src/github.com/ava-labs/gecko/keys/staker.key', 'mode': 'u=r,go='})
ok: [droplet] => (item={'src': '~/go/src/github.com/ava-labs/gecko/keys/staker.crt', 'mode': 'ugo=r'})

TASK [ava-service : Configure AVA service] ******************************************************************
ok: [droplet]

TASK [ava-service : Enable AVA service] *********************************************************************
ok: [droplet]

PLAY RECAP **************************************************************************************************
droplet                    : ok=16   changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

TODO

  • Install under /usr/local

This playbook

 - Installs Gecko dependencies
 - Clones & builds ava-build/gecko
 - Creates an ava user
 - Installs Gecko in /usr/bin
 - Creates and installs a staking certificate
 - Installs Gecko as a Systemd service called "ava"
 - Configures /var/lib/ava/db as the database
 - Configures /var/log/ava as the log destination
 - Starts the service
@moreati
Copy link
Contributor Author

moreati commented May 9, 2020

# ps -u ava -f
UID          PID    PPID  C STIME TTY          TIME CMD
ava       121344       1  0 21:15 ?        00:00:12 /usr/bin/ava --public-ip=178.62.34.87 --db-dir=/var/l
ava       121407  121344  0 21:16 ?        00:00:00 sh -c /usr/lib/ava/plugins/evm
ava       121408  121407  0 21:16 ?        00:00:01 /usr/lib/ava/plugins/evm
# systemctl status ava -l | cat
● ava.service - AVA test node
     Loaded: loaded (/etc/systemd/system/ava.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2020-05-09 21:15:58 UTC; 57min ago
       Docs: https://docs.ava.network/
   Main PID: 121344 (ava)
      Tasks: 18 (limit: 1137)
     Memory: 87.8M
     CGroup: /system.slice/ava.service
             ├─121344 /usr/bin/ava --public-ip=178.62.34.87 --db-dir=/var/lib/ava/db --plugin-dir=/usr/lib/ava/plugins --log-dir=/var/log/ava --log-level=info
             ├─121407 sh -c /usr/lib/ava/plugins/evm
             └─121408 /usr/lib/ava/plugins/evm

May 09 22:12:22 ubuntu-s-1vcpu-1gb-lon1-01 ava[121344]: 2020-05-09 22:12:22.261200 [net info] a28556eb8b: ping from <Conn fd=50 addr=<NetAddr 18.234.153.22:21001> mode=active>
May 09 22:12:25 ubuntu-s-1vcpu-1gb-lon1-01 ava[121344]: 2020-05-09 22:12:25.920303 [net info] a28556eb8b: ping from <Conn fd=52 addr=<NetAddr 107.23.241.199:21001> mode=active>
May 09 22:12:31 ubuntu-s-1vcpu-1gb-lon1-01 ava[121344]: 2020-05-09 22:12:31.059780 [net info] a28556eb8b: ping from <Conn fd=54 addr=<NetAddr 54.197.215.186:21001> mode=active>
May 09 22:12:33 ubuntu-s-1vcpu-1gb-lon1-01 ava[121344]: 2020-05-09 22:12:33.707433 [net info] a28556eb8b: ping from <Conn fd=48 addr=<NetAddr 34.207.133.167:21001> mode=active>
May 09 22:12:40 ubuntu-s-1vcpu-1gb-lon1-01 ava[121344]: 2020-05-09 22:12:40.002157 [net info] a28556eb8b: ping from <Conn fd=46 addr=<NetAddr 3.227.207.132:21001> mode=active>
May 09 22:13:05 ubuntu-s-1vcpu-1gb-lon1-01 ava[121344]: 2020-05-09 22:13:05.259493 [net info] a28556eb8b: ping from <Conn fd=50 addr=<NetAddr 18.234.153.22:21001> mode=active>
May 09 22:13:06 ubuntu-s-1vcpu-1gb-lon1-01 ava[121344]: 2020-05-09 22:13:06.699653 [net info] a28556eb8b: ping from <Conn fd=52 addr=<NetAddr 107.23.241.199:21001> mode=active>
May 09 22:13:07 ubuntu-s-1vcpu-1gb-lon1-01 ava[121344]: 2020-05-09 22:13:07.105992 [net info] a28556eb8b: ping from <Conn fd=54 addr=<NetAddr 54.197.215.186:21001> mode=active>
May 09 22:13:08 ubuntu-s-1vcpu-1gb-lon1-01 ava[121344]: 2020-05-09 22:13:08.807134 [net info] a28556eb8b: ping from <Conn fd=48 addr=<NetAddr 34.207.133.167:21001> mode=active>
May 09 22:13:22 ubuntu-s-1vcpu-1gb-lon1-01 ava[121344]: 2020-05-09 22:13:22.377751 [net info] a28556eb8b: ping from <Conn fd=46 addr=<NetAddr 3.227.207.132:21001> mode=active>

@codecov
Copy link

codecov bot commented May 9, 2020

Codecov Report

Merging #151 into master will decrease coverage by 0.88094%.
The diff coverage is n/a.

@@                 Coverage Diff                 @@
##              master        #151         +/-   ##
===================================================
- Coverage   61.05467%   60.17373%   -0.88095%     
===================================================
  Files            253         246          -7     
  Lines          17029       16577        -452     
===================================================
- Hits           10397        9975        -422     
+ Misses          5734        5695         -39     
- Partials         898         907          +9     

@moreati
Copy link
Contributor Author

moreati commented May 9, 2020

# find /var/lib/ava/
/var/lib/ava/
/var/lib/ava/keys
/var/lib/ava/keys/staker.crt
/var/lib/ava/keys/staker.key
/var/lib/ava/db
/var/lib/ava/db/cascade
/var/lib/ava/db/cascade/v0.2.0
/var/lib/ava/db/cascade/v0.2.0/000022.ldb
/var/lib/ava/db/cascade/v0.2.0/LOG
/var/lib/ava/db/cascade/v0.2.0/MANIFEST-000021
/var/lib/ava/db/cascade/v0.2.0/CURRENT
/var/lib/ava/db/cascade/v0.2.0/CURRENT.bak
/var/lib/ava/db/cascade/v0.2.0/000023.ldb
/var/lib/ava/db/cascade/v0.2.0/000019.ldb
/var/lib/ava/db/cascade/v0.2.0/000020.log
/var/lib/ava/db/cascade/v0.2.0/LOCK
# find /var/log/ava/
/var/log/ava/
/var/log/ava/node
/var/log/ava/node/0.log
/var/log/ava/node/chain
/var/log/ava/node/chain/pUASyygGd8XyCgzeP9KC1GAiZ4Qjycq6fs2HfphA6vSFbvLe6
/var/log/ava/node/chain/pUASyygGd8XyCgzeP9KC1GAiZ4Qjycq6fs2HfphA6vSFbvLe6/0.log
/var/log/ava/node/chain/pUASyygGd8XyCgzeP9KC1GAiZ4Qjycq6fs2HfphA6vSFbvLe6/http
/var/log/ava/node/chain/pUASyygGd8XyCgzeP9KC1GAiZ4Qjycq6fs2HfphA6vSFbvLe6/http/0.log
/var/log/ava/node/chain/2kKSkL4vgVR6QSknoKprsGHVHiEbf7WiGEeSuJUGoB3WzeBhce
/var/log/ava/node/chain/2kKSkL4vgVR6QSknoKprsGHVHiEbf7WiGEeSuJUGoB3WzeBhce/0.log
/var/log/ava/node/chain/2kKSkL4vgVR6QSknoKprsGHVHiEbf7WiGEeSuJUGoB3WzeBhce/http
/var/log/ava/node/chain/2kKSkL4vgVR6QSknoKprsGHVHiEbf7WiGEeSuJUGoB3WzeBhce/http/0.log
/var/log/ava/node/chain/q8CVPqtVEXanDTuU1GnPngLRXVW1WD7XTQi8JVVaJsXgnZDVW
/var/log/ava/node/chain/q8CVPqtVEXanDTuU1GnPngLRXVW1WD7XTQi8JVVaJsXgnZDVW/0.log
/var/log/ava/node/chain/q8CVPqtVEXanDTuU1GnPngLRXVW1WD7XTQi8JVVaJsXgnZDVW/http
/var/log/ava/node/chain/q8CVPqtVEXanDTuU1GnPngLRXVW1WD7XTQi8JVVaJsXgnZDVW/http/0.log
/var/log/ava/node/chain/2mUYSXfLrDtigwbzj1LxKVsHwELghc5sisoXrzJwLqAAQHF4i
/var/log/ava/node/chain/2mUYSXfLrDtigwbzj1LxKVsHwELghc5sisoXrzJwLqAAQHF4i/0.log
/var/log/ava/node/chain/2mUYSXfLrDtigwbzj1LxKVsHwELghc5sisoXrzJwLqAAQHF4i/http
/var/log/ava/node/chain/2mUYSXfLrDtigwbzj1LxKVsHwELghc5sisoXrzJwLqAAQHF4i/http/0.log
/var/log/ava/node/chain/4ktRjsAKxgMr2aEzv9SWmrU7Xk5FniHUrVCX4P1TZSfTLZWFM
/var/log/ava/node/chain/4ktRjsAKxgMr2aEzv9SWmrU7Xk5FniHUrVCX4P1TZSfTLZWFM/0.log
/var/log/ava/node/chain/4ktRjsAKxgMr2aEzv9SWmrU7Xk5FniHUrVCX4P1TZSfTLZWFM/http
/var/log/ava/node/chain/4ktRjsAKxgMr2aEzv9SWmrU7Xk5FniHUrVCX4P1TZSfTLZWFM/http/0.log
/var/log/ava/node/chain/11111111111111111111111111111111LpoYY
/var/log/ava/node/chain/11111111111111111111111111111111LpoYY/0.log
/var/log/ava/node/chain/11111111111111111111111111111111LpoYY/http
/var/log/ava/node/chain/11111111111111111111111111111111LpoYY/http/0.log
/var/log/ava/node/http
/var/log/ava/node/http/0.log

@danlaine
Copy link

@moreati Hey, thanks for making this PR. We haven't forgotten it, we're just really busy 😅. Thanks for your patience

@moreati
Copy link
Contributor Author

moreati commented May 12, 2020 via email

@moreati
Copy link
Contributor Author

moreati commented May 12, 2020

Example upgrade
± ansible-playbook -i droplet.yml service_playbook.yml --diff

PLAY [Configure AVA service] ****************************************************************************

TASK [Gathering Facts] **********************************************************************************
ok: [droplet]

TASK [ava-base : Install deps] **************************************************************************
ok: [droplet]

TASK [gopath : Set GOPATH] ******************************************************************************
ok: [droplet]

TASK [ava-build : Update git clone] *********************************************************************
diff --git a/api/keystore/service.go b/api/keystore/service.go
index 7ca34b5..fe1b349 100644
--- a/api/keystore/service.go
+++ b/api/keystore/service.go
@@ -301,6 +301,70 @@ func (ks *Keystore) ImportUser(r *http.Request, args *ImportUserArgs, reply *Imp
 	return nil
 }
 
+// DeleteUserArgs are arguments for passing into DeleteUser requests
+type DeleteUserArgs struct {
+	Username string `json:"username"`
+	Password string `json:"password"`
...
changed: [droplet]

TASK [ava-build : Build project] ************************************************************************
changed: [droplet]

TASK [ava-user : Create AVA daemon group] ***************************************************************
ok: [droplet]

TASK [ava-user : Create AVA daemon user] ****************************************************************
[WARNING]: The value False (type bool) in a string field was converted to 'False' (type string). If this
does not look like what you expect, quote the entire value to ensure it does not change.
ok: [droplet]

TASK [ava-install : Create directories] *****************************************************************
ok: [droplet] => (item=/var/lib/ava)
--- before
+++ after
@@ -1,7 +1,7 @@
 {
-    "group": 0,
-    "mode": "0755",
-    "owner": 0,
+    "group": 996,
+    "mode": "0500",
+    "owner": 996,
     "path": "/var/lib/ava/staking",
-    "state": "absent"
+    "state": "directory"
 }

changed: [droplet] => (item=/var/lib/ava/staking)
ok: [droplet] => (item=/var/log/ava)
ok: [droplet] => (item=/usr/lib/ava/plugins)

TASK [ava-install : Install binary] *********************************************************************
changed: [droplet]

TASK [ava-install : Install plugins] ********************************************************************
changed: [droplet] => (item={'path': '~/go/src/github.com/ava-labs/gecko/build/plugins/evm'})

TASK [ava-upgrade : Check for Gecko 0.2.0 staking key] **************************************************
ok: [droplet]

TASK [ava-upgrade : Check for Gecko newer staking key] **************************************************
ok: [droplet]

TASK [ava-upgrade : Move staking key] *******************************************************************
changed: [droplet]

TASK [ava-upgrade : Check for Gecko 0.2.0 staking certificate] ******************************************
ok: [droplet]

TASK [ava-upgrade : Check for Gecko newer staking certificate] ******************************************
ok: [droplet]

TASK [ava-upgrade : Migrate staking certificate] ********************************************************
changed: [droplet]

TASK [ava-service : Configure AVA service] **************************************************************
--- before: /etc/systemd/system/ava.service
+++ after: /home/alex/.ansible/tmp/ansible-local-13995rodgh9k8/tmp6w9f48ls/ava.service
@@ -17,7 +17,9 @@
             --db-dir="/var/lib/ava/db" \
             --plugin-dir="/usr/lib/ava/plugins" \
             --log-dir="/var/log/ava" \
-            --log-level="info"
+            --log-level="info" \
+            --staking-tls-cert-file="/var/lib/ava/staking/staker.crt" \
+            --staking-tls-key-file="/var/lib/ava/staking/staker.key"
 
 [Install]
 WantedBy=multi-user.target

changed: [droplet]

TASK [ava-service : Enable AVA service] *****************************************************************
ok: [droplet]

RUNNING HANDLER [ava-service : Reload systemd] **********************************************************
ok: [droplet]

RUNNING HANDLER [ava-service : Restart AVA service] *****************************************************
changed: [droplet]

PLAY RECAP **********************************************************************************************
droplet                    : ok=20   changed=9    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Copy link
Contributor

@holisticode holisticode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moreati thank you very much for this wonderful PR. We appreciate your help in making our product better. In fact configuring ansible to run ava via systemd was in our plans so your work is very welcomed.

I recently joined the team and I will be in charge to take over continuous integration and delivery. My experience with ansible is still limited so I would appreciate you addressing my - for now non-authoritative - comments in order to better understand how this fits into our setup. Some of the comments are meant merely for my colleagues in order to draw attention on a potential issue.

So yet again - thank you!

transport = ssh
deprecation_warnings = false
host_key_checking = false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really want this at the global config level?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to remove it, but note I haven't changed the setting from what is master (as ssh_args = ... o StrictHostKeyChecking=no), merely the method by which it's configured.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right.

-o ControlMaster=auto
-o ControlPersist=60s
-o ForwardAgent=yes
-o UserKnownHostsFile=/dev/null
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about this entry either at the global level?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, I haven't added UserKnownHostsFile=/dev/null, I just happened to reformat that line.

Perhaps your concern is that this ansible.cfg has any effect outside this directory? It is only in effect when this directory is the working directory.

- curl
- g++
- golang-go # Assumes Ubuntu 20.04, where this installs Go 1.13
- libssl-dev
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: These work only on ubuntu.

- cmake
- curl
- g++
- golang-go # Assumes Ubuntu 20.04, where this installs Go 1.13
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we safely make this config choice of Ubuntu 20.04 already?

scripts/ansible/roles/ava-install/defaults/main.yml Outdated Show resolved Hide resolved
@@ -0,0 +1,8 @@
ava_daemon_bin_dir: "/usr/bin"
ava_daemon_data_dir: "/var/lib/{{ ava_daemon_user }}"
ava_daemon_db_dir: "{{ ava_daemon_data_dir }}/db"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@StephenButtolph just recently had defined that the data dir would be ~/.gecko. But maybe for ansible deployments this is fine here?

In any case, note that due to ava_daemon_user: ava, this means that only one user will be able to run ava on such a deployment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would advocate that having the directory /var/lib/ava/.gecko/{staking,db} is not what a user would expect`. Hence why I overrode it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth pointing out that ava_daemon_user: ava and others are only default values. Ansible is very flexible about overriding variables. For instance one could do

- hosts: localhost
  become: true
  roles:
    - {name: ava-user, vars: {ava_daemon_user: snoopy}
    - {name: ava-user, vars: {ava_daemon_user: woodstock}

@tyler-smith
Copy link
Contributor

Hey @moreati thanks for putting this together, it looks great! I had some concerns but most of them appear to have all been addressed. You have "Install under /usr/local" listed as a todo, is this still in progress? Do you have any other changes you're planning on for this PR?

I would really like to split the OS and OS-version (Ubuntu/20.x) requirements out by putting them in their own files and including them conditionally based on the platform, but this could also be done as a follow up PR IMO. @holisticode, I'll let you make the call on that since you're using this right now and I'm not sure if the hard Ubuntu 20.x requirement works for you or not.

@holisticode
Copy link
Contributor

If there is no general opposition, I myself am in favor of using ubuntu 20.x right away because it ships right away with a go version which gecko can run with. Otherwise we need manual installation.

@tyler-smith not sure what you mean by conditionally based on platform, the platform is ubuntu and we use these ansible to just deploy our own nodes, so why would we conditionally support 18.x and 20.x instances?

@moreati
Copy link
Contributor Author

moreati commented May 21, 2020

You have "Install under /usr/local" listed as a todo, is this still in progress?

Yes, probably this weekend

Do you have any other changes you're planning on for this PR?

Not that I can think of. It's already large enough.

This allows them to be run as scripts. They take the same options and
arguments as ansible-playbook.
This makes it easier to skip, if Go has been installed by other means.
Following discussion in ava-labs#151 it
was decided that /usr (and by implication /var) should be reserved for
OS package managers (e.g. apt, yum).
@moreati
Copy link
Contributor Author

moreati commented May 24, 2020

Do you have any other changes you're planning on for this PR?

Not that I can think of. It's already large enough.

I added support for (and defaulted to) --http-host="localhost"

@moreati
Copy link
Contributor Author

moreati commented May 30, 2020

To remove any doubt: I believe I've addressed all comments, and this is ready for review.

aaronbuchwald pushed a commit to aaronbuchwald/avalanchego that referenced this pull request Jul 8, 2020
Added some optimistic logic to prevent un-needed traversals.
Copy link
Contributor

@holisticode holisticode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moreati thank you so much for this wonderful PR, sorry for the looooong delay.

We are going to merge this.
However, it is now slightly out of date - for example, we moved away from salticidae and thus some dependencies got obsolete.

Also, we are committed to support ubuntu 18.04 LTS, while this PR only works with 20.04. Also, we will want to support rpm platforms, and thus the installation commands would be failing there.

And the migrating certs scripts for 0.2.0 we should maybe not even bother by now.

Nevertheless we will not burden this PR to address these things, I made myself a note and we will integrate fixes in follow-up PRs.

In short: thank you for this great contribution, highly appreciated!
And...further ones always welcomed, of course.

@moreati
Copy link
Contributor Author

moreati commented Jul 13, 2020

Thanks, I'll take a look at the other requirements this weekend. I want to add Raspberry Pi OS as well.

@holisticode holisticode merged commit 578cc77 into ava-labs:master Jul 14, 2020
@moreati moreati deleted the ansible-service branch July 20, 2020 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants