Dev: bootstrap: use ssh agent by default #1728

nicholasyang2022 · 2025-03-26T08:52:30Z

ssh-copy-id uses public keys from ssh-agent by default if available. This pull request changes the behavior of ssh bootstrapping in crmsh to keep consistent with ssh-copy-id.

Changes

Option --use-ssh-agent is changed to default ON, and users can opt out with --no-use-ssh-agent.
crmsh will not report an error when environment variable SSH_AUTH_SOCK is not set, or the agent provides no keys. Instead, it fallbacks to using or generating local key files. And no_generating_ssh_keys is removed from crm.conf.
crmsh will always relay environment SSH_AUTH_SOCK accross su/sudo boundary. (Previous behavior is not to relay if no_generatring_ssh_keys is set.)

Close #1633.

codecov · 2025-04-23T05:56:41Z

Codecov Report

Attention: Patch coverage is 71.51163% with 49 lines in your changes missing coverage. Please review.

Project coverage is 70.15%. Comparing base (cafe0a2) to head (1f72896).
Report is 33 commits behind head on master.

Files with missing lines	Patch %	Lines
crmsh/bootstrap.py	72.99%	37 Missing ⚠️
crmsh/ssh_key.py	52.17%	11 Missing ⚠️
crmsh/prun/runner.py	50.00%	1 Missing ⚠️

Additional details and impacted files

Flag	Coverage Δ
integration	`54.37% <62.79%> (?)`
unit	`52.79% <45.34%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crmsh/config.py	`81.48% <ø> (+6.39%)`	⬆️
crmsh/prun/prun.py	`93.87% <100.00%> (+49.65%)`	⬆️
crmsh/report/core.py	`93.69% <ø> (+9.03%)`	⬆️
crmsh/sh.py	`88.88% <100.00%> (+20.55%)`	⬆️
crmsh/ui_cluster.py	`74.81% <100.00%> (+46.05%)`	⬆️
crmsh/user_of_host.py	`97.80% <100.00%> (+61.63%)`	⬆️
crmsh/utils.py	`66.82% <ø> (+18.67%)`	⬆️
crmsh/prun/runner.py	`64.60% <50.00%> (+31.26%)`	⬆️
crmsh/ssh_key.py	`68.65% <52.17%> (+35.32%)`	⬆️
crmsh/bootstrap.py	`86.56% <72.99%> (+38.91%)`	⬆️

... and 47 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

nicholasyang2022 · 2025-04-23T09:09:55Z

Use Cases

Bootstrapping with ssh-agent

> sudo --preserve-env=SSH_AUTH_SOCK crm cluster init -y
INFO: Loading "default" profile from /etc/crm/profiles.yml
INFO: Loading "knet-default" profile from /etc/crm/profiles.yml
WARNING: chronyd.service is not configured to start at system boot.
INFO: Using public keys from ssh-agent...
INFO: Adding public keys to authorized_keys for user root...
INFO: Added public key 256 SHA256:IcI6lesU2A144nILRmIRIOdME2fLB68s3eQrTiyR2NI nicholas@ld0 (ED25519).
INFO: A new ssh keypair is generated for user hacluster.
INFO: A public key is added to authorized_keys for user hacluster: 3072 SHA256:/XivxrQdawkyEbug1D4n1VZIp67xBYnfPDiUZ7lbB1c Cluster internal on ha-2-1 (RSA)
INFO: Configuring csync2
INFO: Starting csync2.socket service on ha-2-1
INFO: BEGIN csync2 checking files
INFO: END csync2 checking files
INFO: Configuring corosync(knet)
WARNING: Not configuring SBD - STONITH will be disabled.
WARNING: Hawk not installed - not configuring web management interface.
WARNING: You should change the hacluster password to something more secure!
INFO: Starting pacemaker.service on ha-2-1
INFO: BEGIN Waiting for cluster
...........                                                                                           INFO: END Waiting for cluster
INFO: Loading initial cluster configuration
INFO: Done (log saved to /var/log/crmsh/crmsh.log on ha-2-1)

> sudo --preserve-env=SSH_AUTH_SOCK crm cluster join -c suse@ha-2-1 -y
WARNING: chronyd.service is not configured to start at system boot.
INFO: Using public keys from ssh-agent...
INFO: A new ssh keypair is generated for user hacluster.
INFO: A public key is added to authorized_keys for user hacluster: 3072 SHA256:C2g5+mZ2Yemg/gJws7tUAghA+be0zwk/zbdhqKpKtZA Cluster internal on ha-2-2 (RSA)
INFO: Configuring csync2
INFO: Starting csync2.socket service
INFO: BEGIN csync2 syncing files in cluster
INFO: END csync2 syncing files in cluster
INFO: Merging known_hosts
Warning: Permanently added 'ha-2-2' (ED25519) to the list of known hosts.
INFO: BEGIN Probing for new partitions
INFO: END Probing for new partitions

WARNING: Hawk not installed - not configuring web management interface.
WARNING: You should change the hacluster password to something more secure!
INFO: Starting pacemaker.service on ha-2-2
INFO: BEGIN Waiting for cluster
..                                                                                                    INFO: END Waiting for cluster
INFO: Set property "priority" in rsc_defaults to 1
INFO: BEGIN Reloading cluster configuration
INFO: END Reloading cluster configuration
INFO: Done (log saved to /var/log/crmsh/crmsh.log on ha-2-2)

Bootstrapping without ssh-agent

> sudo crm cluster init -y
INFO: Loading "default" profile from /etc/crm/profiles.yml
INFO: Loading "knet-default" profile from /etc/crm/profiles.yml
WARNING: chronyd.service is not configured to start at system boot.
INFO: A new ssh keypair is generated for user root.
INFO: A public key is added to authorized_keys for user root: 3072 SHA256:Zy4BVpNpxO4mDv68G+N02/063w74Z/n6H4joNzbn3WA Cluster internal on ha-2-1 (RSA)
INFO: A new ssh keypair is generated for user hacluster.
INFO: A public key is added to authorized_keys for user hacluster: 3072 SHA256:t3iXoqgYnimXcZ9OAjR3GQnLKOwy8YPII/F0ugELwKM Cluster internal on ha-2-1 (RSA)
INFO: Configuring csync2
INFO: Starting csync2.socket service on ha-2-1
INFO: BEGIN csync2 checking files
INFO: END csync2 checking files
INFO: Configuring corosync(knet)
WARNING: Not configuring SBD - STONITH will be disabled.
WARNING: Hawk not installed - not configuring web management interface.
WARNING: You should change the hacluster password to something more secure!
INFO: Starting pacemaker.service on ha-2-1
INFO: BEGIN Waiting for cluster
...........                                                                                           INFO: END Waiting for cluster
INFO: Loading initial cluster configuration
INFO: Done (log saved to /var/log/crmsh/crmsh.log on ha-2-1)

> sudo crm cluster join -c suse@ha-2-1 -y
WARNING: chronyd.service is not configured to start at system boot.
INFO: A new ssh keypair is generated for user suse.
INFO: A public key is added to authorized_keys for user suse: 3072 SHA256:6nLsGwbzVTPheyQ7OOc29p4NXOIUXvhXKKaFWOlzXys Cluster internal on ha-2-2 (RSA)
INFO: Configuring SSH passwordless with suse@ha-2-1
(suse@ha-2-1) Password: 
INFO: A public key is added to authorized_keys for user suse: 3072 SHA256:3QkmL7w8J/0VOEnJQqimnMIb4u8ir+sHAH3XhXs9Be0 Cluster internal on ha-2-1 (RSA)
INFO: A new ssh keypair is generated for user hacluster.
INFO: A public key is added to authorized_keys for user hacluster: 3072 SHA256:Iy4nI8+lGN55wZaVYK+UmwIS2nEAxU63M/uAjCNCj/U Cluster internal on ha-2-2 (RSA)
INFO: Configuring csync2
INFO: Starting csync2.socket service
INFO: BEGIN csync2 syncing files in cluster
INFO: END csync2 syncing files in cluster
INFO: Merging known_hosts
Warning: Permanently added 'ha-2-2' (ED25519) to the list of known hosts.
INFO: BEGIN Probing for new partitions
INFO: END Probing for new partitions

WARNING: Hawk not installed - not configuring web management interface.
WARNING: You should change the hacluster password to something more secure!
INFO: Starting pacemaker.service on ha-2-2
INFO: BEGIN Waiting for cluster
..                                                                                                    INFO: END Waiting for cluster
INFO: Set property "priority" in rsc_defaults to 1
INFO: BEGIN Reloading cluster configuration
INFO: END Reloading cluster configuration
INFO: Done (log saved to /var/log/crmsh/crmsh.log on ha-2-2)

Bootstrapping with `--no-ssh-agent`

> sudo --preserve-env=SSH_AUTH_SOCK crm cluster init --no-use-ssh-agent -y
INFO: Loading "default" profile from /etc/crm/profiles.yml
INFO: Loading "knet-default" profile from /etc/crm/profiles.yml
WARNING: chronyd.service is not configured to start at system boot.
INFO: A new ssh keypair is generated for user root.
INFO: A public key is added to authorized_keys for user root: 3072 SHA256:5pgSYviLUjW2P6p0xXioVrqbGlcytmiG4S8UfTubHD8 Cluster internal on ha-2-1 (RSA)
INFO: A new ssh keypair is generated for user hacluster.
INFO: A public key is added to authorized_keys for user hacluster: 3072 SHA256:bO0G32gkJm4icHNyBbrRK6IxPuQqrQWb/DHWc1Asldo Cluster internal on ha-2-1 (RSA)
INFO: Configuring csync2
INFO: Starting csync2.socket service on ha-2-1
INFO: BEGIN csync2 checking files
INFO: END csync2 checking files
INFO: Configuring corosync(knet)
WARNING: Not configuring SBD - STONITH will be disabled.
WARNING: Hawk not installed - not configuring web management interface.
WARNING: You should change the hacluster password to something more secure!
INFO: Starting pacemaker.service on ha-2-1
INFO: BEGIN Waiting for cluster
...........                                                                                           INFO: END Waiting for cluster
INFO: Loading initial cluster configuration
INFO: Done (log saved to /var/log/crmsh/crmsh.log on ha-2-1)

> sudo --preserve-env=SSH_AUTH_SOCK crm cluster join -c suse@ha-2-1 --no-use-ssh-agent -y
WARNING: chronyd.service is not configured to start at system boot.
INFO: A new ssh keypair is generated for user suse.
INFO: A public key is added to authorized_keys for user suse: 3072 SHA256:pcrL2J61QWUI/z7iI6lcfEndThWeH1wrf7hJuNMm3xU Cluster internal on ha-2-2 (RSA)
INFO: Configuring SSH passwordless with suse@ha-2-1
(suse@ha-2-1) Password: 
INFO: A public key is added to authorized_keys for user suse: 3072 SHA256:6gERDAE2G9cfqaruUINGIwTv9Kabh4C/8s/FMYkqkoQ Cluster internal on ha-2-1 (RSA)
INFO: A new ssh keypair is generated for user hacluster.
INFO: A public key is added to authorized_keys for user hacluster: 3072 SHA256:D31Sfc6/H0/1pqGeRjq6QP6R2gKtRAwhHNtrX5GF5sk Cluster internal on ha-2-2 (RSA)
INFO: Configuring csync2
INFO: Starting csync2.socket service
INFO: BEGIN csync2 syncing files in cluster
INFO: END csync2 syncing files in cluster
INFO: Merging known_hosts
Warning: Permanently added 'ha-2-2' (ED25519) to the list of known hosts.
INFO: BEGIN Probing for new partitions
INFO: END Probing for new partitions

WARNING: Hawk not installed - not configuring web management interface.
WARNING: You should change the hacluster password to something more secure!
INFO: Starting pacemaker.service on ha-2-2
INFO: BEGIN Waiting for cluster
..                                                                                                    INFO: END Waiting for cluster
INFO: Set property "priority" in rsc_defaults to 1
INFO: BEGIN Reloading cluster configuration
INFO: END Reloading cluster configuration
INFO: Done (log saved to /var/log/crmsh/crmsh.log on ha-2-2)

liangxin1300 · 2025-04-23T14:54:16Z

crmsh/bootstrap.py

        logger.info("Adding public keys to authorized_keys for user %s...", local_user)
        for key in ssh_public_keys:
            authorized_key_manager.add(None, local_user, key)
+            logger.info("Added public key %s.", key.fingerprint())


info level, or debug for fingerprint?

As we allow to use keys from either ssh-agent or local key files, I think it is needed to show which key is actually used in the UI.

and use-ssh-agent by default

…le (ClusterLabs#1663)

…lable (ClusterLabs#1663)

…-agent fails (ClusterLabs#1663)

…agent (ClusterLabs#1633)

… merging authorized_keys (ClusterLabs#1663) As we now allow to use either ssh-agent or local key files (unless --no-use-ssh-agent is specified), we cannot expect there are local key files on remote nodes. As a result, we cannot swap ssh keys by fetching key files from remote. Instead, we swap keys by merging authorized_keys.

…Labs#1633) as we allow to mix agent-based and key file based authentication.

…agent fails (ClusterLabs#1663)

…erLabs#1663)

)

…rLabs#1663) We support using ssh-agent only when keys are already added to authorized_keys. Otherwise, we use ore generate local key files.

zzhou1

Great work! Three use cases verified:

normal 2-node bootstrap
bootstrap qdevice
crm cluster copy

liangxin1300

Thanks for the nice work!

nicholasyang2022 force-pushed the crmsh-1663-use-ssh-agent-by-default branch 3 times, most recently from 81e6dce to f03db3f Compare March 31, 2025 07:49

nicholasyang2022 force-pushed the crmsh-1663-use-ssh-agent-by-default branch from f03db3f to b9ff708 Compare April 7, 2025 09:44

nicholasyang2022 force-pushed the crmsh-1663-use-ssh-agent-by-default branch 4 times, most recently from 7332dff to cf767a3 Compare April 22, 2025 08:05

nicholasyang2022 force-pushed the crmsh-1663-use-ssh-agent-by-default branch 3 times, most recently from 4e1ebb5 to 3f12ab8 Compare April 23, 2025 05:34

nicholasyang2022 force-pushed the crmsh-1663-use-ssh-agent-by-default branch from 9a5f3aa to 9b62c6a Compare April 23, 2025 09:01

nicholasyang2022 marked this pull request as ready for review April 23, 2025 09:28

nicholasyang2022 requested review from liangxin1300 and zzhou1 April 23, 2025 09:28

liangxin1300 reviewed Apr 23, 2025

View reviewed changes

nicholasyang2022 force-pushed the crmsh-1663-use-ssh-agent-by-default branch from 51e1f17 to 413fb76 Compare April 27, 2025 04:45

nicholasyang2022 added 10 commits April 27, 2025 14:11

Dev: ui_cluster: add --no-use-ssh-agent (ClusterLabs#1663)

9059546

and use-ssh-agent by default

Dev: bootstrap init: do not raise an error when no ssh key is availab…

8cfbe00

…le (ClusterLabs#1663)

Dev: bootstrap join: do not raise an error when no ssh key is availab…

4367cfc

…le (ClusterLabs#1663)

Dev: bootstrap init -N: do not raise an error when no ssh key is avai…

6526421

…lable (ClusterLabs#1663)

Dev: bootstrap join: allow to fallback to keyfile when login with ssh…

c67a48d

…-agent fails (ClusterLabs#1663)

Fix: ssh_key: should not use ssh-copy-id -i when using keys from ssh-…

3caf572

…agent (ClusterLabs#1633)

Dev: bootstrap: should pass SSH_AUTH_SOCK when swapping keys (Cluster…

15942a9

…Labs#1633) as we allow to mix agent-based and key file based authentication.

Dev: bootstrap geo: allow to fallback to keyfile when login with ssh-…

f32ec45

…agent fails (ClusterLabs#1663)

Dev: remove use_ssh_ssh_agent() and always relay SSH_AUTH_SOCK (Clust…

697a29f

…erLabs#1663)

nicholasyang2022 force-pushed the crmsh-1663-use-ssh-agent-by-default branch from 413fb76 to 6869846 Compare April 27, 2025 06:11

nicholasyang2022 added 5 commits April 27, 2025 14:13

Dev: user_of_host: remove unused no_generating_ssh_key (ClusterLabs#1663

b241e53

)

Dev: unittest: adjust unit tests for previous changes

4f9525e

Fix: codecov: move sitecustomize hook to python3.13

465b765

Dev: behave: adjust functional tests for previous changes

29db9f1

Dev: boostrap: ssh_copy_id should not use keys from ssh-agent (Cluste…

80b2320

…rLabs#1663) We support using ssh-agent only when keys are already added to authorized_keys. Otherwise, we use ore generate local key files.

nicholasyang2022 force-pushed the crmsh-1663-use-ssh-agent-by-default branch 2 times, most recently from 794c18d to 439f896 Compare April 27, 2025 06:22

zzhou1 approved these changes Apr 27, 2025

View reviewed changes

Dev: bootstrap: minor message wording refine

1f72896

nicholasyang2022 force-pushed the crmsh-1663-use-ssh-agent-by-default branch from 439f896 to 1f72896 Compare April 29, 2025 03:58

liangxin1300 approved these changes Apr 29, 2025

View reviewed changes

liangxin1300 merged commit 3989d08 into ClusterLabs:master May 6, 2025
34 checks passed

nicholasyang2022 deleted the crmsh-1663-use-ssh-agent-by-default branch May 9, 2025 05:07

liangxin1300 mentioned this pull request Sep 28, 2025

Ask password for merging known_hosts on join node when passwordless configured before #1931

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Dev: bootstrap: use ssh agent by default #1728

Dev: bootstrap: use ssh agent by default #1728

Uh oh!

nicholasyang2022 commented Mar 26, 2025 •

edited

Loading

Uh oh!

codecov bot commented Apr 23, 2025 •

edited

Loading

Uh oh!

nicholasyang2022 commented Apr 23, 2025

Uh oh!

liangxin1300 Apr 23, 2025

Uh oh!

nicholasyang2022 Apr 27, 2025

Uh oh!

zzhou1 left a comment

Uh oh!

liangxin1300 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Dev: bootstrap: use ssh agent by default #1728

Dev: bootstrap: use ssh agent by default #1728

Uh oh!

Conversation

nicholasyang2022 commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

codecov bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

nicholasyang2022 commented Apr 23, 2025

Use Cases

Bootstrapping with ssh-agent

Bootstrapping without ssh-agent

Bootstrapping with --no-ssh-agent

Uh oh!

liangxin1300 Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

nicholasyang2022 Apr 27, 2025

Choose a reason for hiding this comment

Uh oh!

zzhou1 left a comment

Choose a reason for hiding this comment

Uh oh!

liangxin1300 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nicholasyang2022 commented Mar 26, 2025 •

edited

Loading

codecov bot commented Apr 23, 2025 •

edited

Loading

Bootstrapping with `--no-ssh-agent`