cannot ssh into nodes

[lghinet]

hello,
i cannot ssh into nodes because my ip has changed

[ktsakalozos]

Hi @lghinet

Would you run this cdk agent: https://github.com/juju-solutions/cdk-field-agent so we get some more info on what the state of the deployment is?

Also could you give us some context on what exactly happened and you cannot ssh anymore on the machines? You said your IP changed, how/why? Is it possible your ssh keys also changed? Are you able to directly ssh to the machines using their IP eg ssh ubuntu@10.1.3.198? You may need to re-importing yous ssh keys to Juju. Have a look at juju import-ssh-key and juju add-ssh-key. Another test you may want to try is to provision a new ubuntu machine juju deploy ubuntu and then try to ssh to that.

Thank you for reaching out.

[lghinet]

my local ubuntu ip changed from 10.1.3.157 to 10.1.3.36
i can access https://10.1.3.36:8443/1.0 but juju remembers the old ip

$ juju ssh-keys
No keys to display.

$ ssh ubuntu@10.1.3.220
The authenticity of host '10.1.3.220 (10.1.3.220)' can't be established.
ECDSA key fingerprint is SHA256:RyXW+zN2ouapH4J2/GIT7cabQ4O/PHEeKTo4+lRMCzk.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.1.3.220' (ECDSA) to the list of known hosts.
Permission denied (publickey).

[ktsakalozos]

So you can ssh to the machines directly but when you ssh via juju juju goes through your old IP.

I haven't seen this before. The issue seems to be related to the localhost/lxd environment and Juju. I do not think you will find anything interesting in juju controller-config -c <your_controller> or in juju --debug ssh <you_unit>

Would you be able to open a bug against juju here: https://bugs.launchpad.net/juju ?

[ktsakalozos]

You might also want to double check that the controller endpoints in ~/.local/share/juju/controllers.yaml are correct.

[lghinet]

bootstrap-params are wrong on the juju controller machine
i change it, restart, nothing

root@juju-815021-0:/var/lib/juju# cat bootstrap-params
controller-config:
  api-port: 17070
  auditing-enabled: false
  
  controller-uuid: 3b4f7e1e-c1d1-46f3-8740-704f59f93fe2
  max-logs-age: 72h
  max-logs-size: 4096M
  max-txn-log-size: 10M
  set-numa-control-policy: false
  state-port: 37017
controller-model-config:
  agent-metadata-url: ""
  agent-stream: released
  agent-version: 2.3-alpha1.1
  apt-ftp-proxy: ""
  apt-http-proxy: ""
  apt-https-proxy: ""
  apt-mirror: ""
  apt-no-proxy: ""
  authorized-keys: |
    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2XXDwBPORHLp62lhAlIbnkLhsYIFKqKrDhvLD3BWDAdsEegZjjGNn+1wfculBDoTRHLpcecqxpJLs6ezM1QrSU4rMMsNkpXun87bQYX6smBjwRw7Cs4gpm/FzAutQgVcTeoC4jbnIA3tbKsqt4qMr4t6tlxMRLsjsFcji04Hl46enpJP8q0LR2o5ltUbLOpjI/YURmEnKO+anBrhWkk1I+ZkPiODVVQlrhcqnL5Zl0rTUgwpGHqoa4KYt7HgH/VSNSB29Q6Pyuy6rGGK40xDPD5Faan6fpQE9vYwu7z9JmSGGT60wJ2nLwqu6+LcPiHBhTilcaHuf5RlVgghZtkVD juju-client-key
  automatically-retry-hooks: true
  default-series: xenial
  development: false
  disable-network-management: false
  egress-subnets: ""
  enable-os-refresh-update: true
  enable-os-upgrade: true
  firewall-mode: instance
  ftp-proxy: ""
  http-proxy: ""
  https-proxy: ""
  ignore-machine-addresses: false
  image-metadata-url: ""
  image-stream: daily
  logforward-enabled: false
  logging-config: <root>=WARNING;unit=DEBUG
  max-action-results-age: 336h
  max-action-results-size: 5G
  max-status-history-age: 336h
  max-status-history-size: 5G
  name: controller
  net-bond-reconfigure-delay: 17
  no-proxy: 127.0.0.1,localhost,::1
  provisioner-harvest-mode: destroyed
  proxy-ssh: false
  resource-tags: {}
  ssl-hostname-verification: true
  test-mode: false
  transmit-vendor-metrics: true
  type: lxd
  update-status-hook-interval: 5m
  uuid: 6de94b93-a2dc-47a5-8402-2f0c33815021
controller-model-version: 0
hosted-model-config:
  image-stream: daily
  name: conjure-canonical-kubern-322
  uuid: 1909d289-f31a-4a94-8278-d5891ec7936c
bootstrap-machine-instance-id: juju-815021-0
bootstrap-machine-constraints:
  mem: 3584
bootstrap-machine-hardware:
  arch: amd64
  mem: 0
  cpucores: 0
model-constraints: {}
custom-image-metadata: "null"
controller-cloud: |
  name: localhost
  type: lxd
  auth-types: [interactive, certificate]
  endpoint: 10.1.3.157:8443
  regions:
    localhost:
      endpoint: 10.1.3.157:8443
controller-cloud-region: localhost
controller-cloud-credential-name: localhost
controller-cloud-credential:

endpoint: 10.1.3.157:8443
regions:
localhost:
endpoint: 10.1.3.157:8443

from above

[ktsakalozos]

I hear you are hitting an issue with the lxd provider upstream. Can you please open a bug against Juju?