Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add more logging to help diagnose upgrade issue #15862

Merged
merged 1 commit into from
Jul 5, 2023
Merged

Add more logging to help diagnose upgrade issue #15862

merged 1 commit into from
Jul 5, 2023

Conversation

SimonRichardson
Copy link
Member

The following adds some timeout + full status whilst it attempts to upgrade. We may need to crack open lxd to see the full server logs to see what's actually happening. For now, see if this helps.

Checklist

QA steps

$ cd tests && ./main.sh -v upgrade test_upgrade_simplestream

I've run this multiple times and the output is the same.

==> Checking for dependencies
==> Using Juju located at /home/simon/go/bin/juju
==> Running subtest test_upgrade_simplestream for upgrade suite
==> TEST BEGIN: upgrade (tmp.H1T)
==> Checking for dependencies
===> [   ] Running: simplestream metadata last stable
===> Using jujud version 3.2.2-ubuntu-amd64
===> Testing against stable version 3.2.0
Finding agent binaries in /home/simon/go/src/github.com/juju/juju/tests/suites/upgrade/streams for stream released.

    | 17:19:30 INFO  juju.cmd supercommand.go:56 running juju [3.2.2 1f8427fb0daa94c31bac35d8cc5c52e12cec11c9 gc go1.20.3]
    | 17:19:30 INFO  cmd cloudcredential.go:47 updating credential store
    | 17:19:31 INFO  cmd authkeys.go:113 Adding contents of "/home/simon/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
    | 17:19:31 INFO  cmd authkeys.go:113 Adding contents of "/home/simon/.ssh/id_ed25519.pub" to authorized-keys
    | Creating Juju controller "test-upgrade-stable-stream" on lxd/default
    | 17:19:31 INFO  juju.cmd.juju.commands bootstrap.go:1026 combined bootstrap constraints:
    | 17:19:31 INFO  cmd bootstrap.go:403 Loading image metadata
    | Looking for packaged Juju agent version 3.2.0 for amd64
    | 17:19:31 INFO  juju.environs.bootstrap tools.go:78 looking for bootstrap agent binaries: version=3.2.0
    | 17:19:31 INFO  juju.environs.bootstrap tools.go:80 found 1 packaged agent binaries
    | Located Juju agent version 3.2.0-ubuntu-amd64 at https://streams.canonical.com/juju/tools/agent/3.2.0/juju-3.2.0-linux-amd64.tgz
    | 17:19:31 INFO  cmd bootstrap.go:580 Starting new instance for initial controller
    | To configure your system to better support LXD containers, please see: https://linuxcontainers.org/lxd/docs/master/explanation/performance_tuning/
    | Launching controller instance(s) on lxd/default...
17:19:31 INFO  juju.container.lxd container.go:294 starting new container "juju-1a89a5-0" (image "")
17:19:34 INFO  juju.provider.lxd environ_broker.go:48 started instance "juju-1a89a5-0"
    | - juju-1a89a5-0 (arch=amd64)
    | 17:19:34 INFO  juju.environs.bootstrap bootstrap.go:1005 newest version: 3.2.0
    | 17:19:34 INFO  juju.environs.bootstrap bootstrap.go:1012 failed to find 3.2.2 agent binaries, will attempt to use 3.2.0
    | 17:19:34 INFO  juju.environs.bootstrap bootstrap.go:1020 picked bootstrap agent binary version: 3.2.0
    | Installing Juju agent on bootstrap instance
    | Waiting for address
    | Attempting to connect to 240.60.0.112:22
    | Connected to 240.60.0.112
    | 17:19:50 INFO  juju.cloudconfig userdatacfg_unix.go:615 Fetching agent: curl -sSf --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/3.2.0/juju-3.2.0-linux-amd64.tgz]>
    | Running machine configuration script...
    | Bootstrap agent now started
    | 17:20:32 INFO  juju.juju api.go:340 API endpoints changed from [] to [240.60.0.112:17070]
    | Contacting Juju controller at 240.60.0.112 to verify accessibility...
    | 17:20:32 INFO  juju.juju api.go:86 connecting to API addresses: [240.60.0.112:17070]
    | 17:20:36 INFO  juju.api apiclient.go:702 connection established to "wss://240.60.0.112:17070/model/8ad7547d-3d62-46a0-8f02-c74da21a89a5/api"

    | Bootstrap complete, controller "test-upgrade-stable-stream" is now available
    | Controller machines are in the "controller" model

    | Now you can run
    | juju add-model <model-name>
    | to create a new model to deploy workloads.
    | 17:20:36 INFO  cmd supercommand.go:555 command finished

Added 'test-upgrade-stable' model on lxd/default with credential 'lxd' for user 'admin'
Located charm "jameinel-ubuntu-lite" in charm-hub, revision 10
Deploying "jameinel-ubuntu-lite" from charm-hub charm "jameinel-ubuntu-lite", revision 10 in channel stable on ubuntu@20.04/stable
[+] (attempt 0) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:20:40+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 1) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:20:45+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 2) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:20:50+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 3) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:20:56+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 4) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:01+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 5) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:06+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 6) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:11+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 7) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:17+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 8) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:22+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 9) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:27+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 10) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:32+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 11) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:37+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 12) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:43+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 13) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:48+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 14) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:53+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04
[+] (attempt 15) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:21:58+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id  Base          AZ  Message
    | 0        pending           pending  ubuntu@20.04      Creating container
[+] (attempt 16) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:22:04+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id        Base          AZ  Message
    | 0        pending           juju-9d0eac-0  ubuntu@20.04      Container started
[+] (attempt 17) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:22:09+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id        Base          AZ  Message
    | 0        pending           juju-9d0eac-0  ubuntu@20.04      Container started
[+] (attempt 18) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:22:14+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0                               waiting for machine
    |
    | Machine  State    Address  Inst id        Base          AZ  Message
    | 0        pending           juju-9d0eac-0  ubuntu@20.04      Container started
[+] (attempt 19) polling status for .applications | select((.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["juju-status"] | .current == "idle") and (.["ubuntu-lite"] | .units | .["ubuntu-lite/0"] | .["workload-status"] | .current != "error")) | keys[0] => ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:22:19+01:00
    |
    | App          Version  Status   Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite           waiting    0/1  jameinel-ubuntu-lite  stable    10  no       waiting for machine
    |
    | Unit           Workload  Agent       Machine  Public address  Ports  Message
    | ubuntu-lite/0  waiting   allocating  0        240.60.0.61            waiting for machine
    |
    | Machine  State    Address      Inst id        Base          AZ  Message
    | 0        pending  240.60.0.61  juju-9d0eac-0  ubuntu@20.04      Running
[+] Completed polling status for ubuntu-lite
    | Model                Controller                  Cloud/Region  Version  SLA          Timestamp
    | test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.0    unsupported  17:22:25+01:00
    |
    | App          Version  Status  Scale  Charm                 Channel  Rev  Exposed  Message
    | ubuntu-lite  20.04    active      1  jameinel-ubuntu-lite  stable    10  no       ready
    |
    | Unit            Workload  Agent  Machine  Public address  Ports  Message
    | ubuntu-lite/0*  active    idle   0        240.60.0.61            ready
    |
    | Machine  State    Address      Inst id        Base          AZ  Message
    | 0        started  240.60.0.61  juju-9d0eac-0  ubuntu@20.04      Running
==> Current juju version 3.2.0
best version:
    3.2.2
started upgrade to 3.2.2
[+] (attempt 0) polling machines
Model       Controller                  Cloud/Region  Version  SLA          Timestamp
controller  test-upgrade-stable-stream  lxd/default   3.2.2    unsupported  17:22:30+01:00

App         Version  Status  Scale  Charm            Channel     Rev  Exposed  Message
controller           active      1  juju-controller  3.2/stable   14  no

Unit           Workload  Agent  Machine  Public address  Ports  Message
controller/0*  active    idle   0        240.60.0.112

Machine  State    Address       Inst id        Base          AZ  Message
0        started  240.60.0.112  juju-1a89a5-0  ubuntu@22.04      Running
test-upgrade-stable-stream:admin/test-upgrade-stable (no change)
best version:
    3.2.2
started upgrade to 3.2.2
[+] (attempt 1) polling machines
Model                Controller                  Cloud/Region  Version  SLA          Timestamp
test-upgrade-stable  test-upgrade-stable-stream  lxd/default   3.2.2    unsupported  17:22:51+01:00

App          Version  Status  Scale  Charm                 Channel  Rev  Exposed  Message
ubuntu-lite  20.04    active      1  jameinel-ubuntu-lite  stable    10  no       ready

Unit            Workload  Agent  Machine  Public address  Ports  Message
ubuntu-lite/0*  active    idle   0        240.60.0.61            ready

Machine  State    Address      Inst id        Base          AZ  Message
0        started  240.60.0.61  juju-9d0eac-0  ubuntu@20.04      Running
===> [ ✔ ] Success: simplestream metadata last stable (233s)
suites/upgrade/streams.sh: line 1: 2941430 Killed                  tail -f "${TEST_DIR}/${TEST_CURRENT}.log" 2> /dev/null
==> SKIP: Asked to skip stream (previous) tests
==> TEST DONE: upgrade (233s)
==> Cleaning up
====> Cleaning up jujus
====> Introspection gathering
====> Introspection gathered
====> Removing offers
====> Removed offers
====> Destroying juju (test-upgrade-stable-stream)

    | WARNING This command will destroy the "test-upgrade-stable-stream" controller and all its resources
    | Destroying controller
    | Waiting for model resources to be reclaimed
    | Waiting for 1 model, 1 machine, 2 applications
    | Waiting for 1 model, 1 machine
    | Waiting for 1 model, 1 machine
    | Waiting for 1 model, 1 machine
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | Waiting for 1 model
    | All models reclaimed, cleaning up controller machines

====> Destroyed juju (test-upgrade-stable-stream)
====> Completed cleaning up jujus
====> Running clean up func: remove_upgrade_tools
==> Removing tools
==> Removed tools
====> Finished cleaning up func: remove_upgrade_tools
====> Running clean up func: remove_upgrade_metadata
==> Removing metadata
==> Removed metadata
====> Finished cleaning up func: remove_upgrade_metadata
====> Running clean up func: kill_server
==> Killing server
==> Killed server (PID is 2941572)
====> Finished cleaning up func: kill_server

==> Test result: success
==> Tests Removed: /home/simon/go/src/github.com/juju/juju/tests/tmp.H1T
==> TEST COMPLETE

@SimonRichardson
Add more logging to help diagnose upgrade issue
07db619 
The following adds some timeout + full status whilst it attempts
to upgrade. We may need to crack open lxd to see the full server logs
to see what's actually happening. For now, see if this helps.
@SimonRichardson SimonRichardson self-assigned this Jul 5, 2023
@SimonRichardson
Copy link
Member Author

/merge

@jujubot jujubot merged commit 9f1a575 into juju:3.2 Jul 5, 2023
6 of 7 checks passed
@SimonRichardson SimonRichardson deleted the add-more-logging-to-help-diagnose-issue branch July 5, 2023 19:09
This was referenced Jul 6, 2023
Closed
Merged
jujubot added a commit that referenced this pull request Jul 7, 2023
@jujubot
Merge pull request #15881 from jack-w-shaw/3.2-into-3.3
87d8ddc 
#15881

Merges:
- #15845
- #15852
- #15854
- #15849
- #15855
- #15857
- #15861
- #15862
- #15825
- #15866
- #15863
- #15870
- #15871
- #15872
- #15873
- #15874
- #15875

One substantial conflict in `cmd/juju/application/deployer/bundlehandler.go` & `cmd/juju/application/deployer/bundlehandler_test.go`, resulting in failing unit tests.

The conflict relates to this PR:
- #15825

These were fixed by the two tops commits

### QA Steps

Ensure all unit tests pass
```
go test github.com/juju/juju/cmd/juju/application/deployer
go test github.com/juju/juju/core/charm
go test github.com/juju/juju/apiserver/facades/client/application
```

Ensure we can deploy a complex bundle
```
juju deploy kubeflow
```
@hpidcock hpidcock mentioned this pull request Jul 11, 2023
Merged
jujubot added a commit that referenced this pull request Jul 11, 2023
@jujubot
Merge pull request #15896 from hpidcock/merge-3.3-main-20230711
ed59a7a 
#15896

Forward ports:
- #15845
- #15725
- #15852
- #15854
- #15849
- #15855
- #15857
- #15861
- #15862
- #15860
- #15864
- #15825
- #15866
- #15863
- #15870
- #15871
- #15872
- #15873
- #15874
- #15876
- #15875
- #15881
- #15727
- #15883
- #15884
- #15880
- #15879
- #15886
- #15887
- #15877
- #15888
- #15893
- #15894

Conflicts:
- cmd/juju/ssh/debugcode_test.go
- cmd/juju/ssh/debughooks_test.go
- cmd/juju/ssh/scp_unix_test.go
- cmd/juju/ssh/ssh_machine.go
- cmd/juju/ssh/ssh_machine_test.go
- cmd/juju/ssh/ssh_unix_test.go
- worker/dbaccessor/worker.go
- caas/kubernetes/provider/k8s_test.go
- caas/kubernetes/provider/operator_test.go
- caas/kubernetes/provider/package_test.go
- caas/kubernetes/provider/rbac.go
- cmd/juju/application/deployer/bundlehandler.go
- cmd/juju/application/deployer/bundlehandler_test.go
- cmd/jujud/agent/model/manifolds.go
- core/bundle/changes/changes.go
- worker/caasapplicationprovisioner/application.go
- worker/caasapplicationprovisioner/application_test.go
- worker/caasapplicationprovisioner/mock_test.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anvial anvial anvial approved these changes

@hpidcock hpidcock Awaiting requested review from hpidcock

Assignees

@SimonRichardson SimonRichardson

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@SimonRichardson @anvial @jujubot