Migrate docker container from ubuntu 20.04 to 22.04 #5877

rveerama1 · 2023-10-25T11:10:10Z

Migration attempt has been done few times already in following PRs. #5072, #5449, #5456, #5487 and dropped due to various reasons. One main reason was failure of the test named live_migration::live_migration_sequential::test_live_migration_ovs_dpdk #5532.

After analyzing those PRs I would like to approach this issue step by step.

Migrate to Ubuntu 22.04 with minimal changes where all the tests will be passed and CI will be happy.
Upgrade SPDK to latest or stable version.
Upgrade virtiofsd to latest or stable version.
Upgrade any other dependencies to latest or stable version based on how CI proceeds with tests or any other specific requirements.

@likebreath @rbradford @michael2012z Any comments or suggestions?

likebreath · 2023-10-25T18:13:15Z

@rveerama1 Thank you for looking into this. I am glad to see that you are learning from previous attempts.

In general, I'd suggest you focus on x86_64 worker first, and gradually move to other workers, say aarch64, amd, windows workers. On the other hand, I believe most of the debugging work can be done offline, either on your own machine or a equivalent Azure VM instance. I'd only use the CI pipeline to validate once a certain worker is working locally.

likebreath · 2023-10-25T18:18:18Z

Instead of using this PR, let's have the discussion and track the process in the issue #5878.

rveerama1 · 2023-10-26T08:14:33Z

In general, I'd suggest you focus on x86_64 worker first, and gradually move to other workers, say aarch64, amd, windows workers.

Sure.

rveerama1 · 2023-10-27T15:55:04Z

some progress with the tests

just migrating ubuntu from 20.04 to 22.04 and keeping the virtiofsd to 1.1.0 version caused below tests to fail.

[2023-10-25T11:35:44.202Z] cloud-hypervisor: 7.164748s: <vmm> INFO:vmm/src/lib.rs:1952 -- API request event: VmAddFs(FsConfig { tag: "myfs", socket: "/tmp/ch8QW0js/virtiofs.sock", num_queues: 1, queue_size: 1024, id: Some("myfs0"), pci_segment: 15 }, Sender { .. })
[2023-10-25T11:35:44.202Z] cloud-hypervisor: 7.164813s: <vmm> INFO:vmm/src/device_manager.rs:2614 -- Creating virtio-fs device: FsConfig { tag: "myfs", socket: "/tmp/ch8QW0js/virtiofs.sock", num_queues: 1, queue_size: 1024, id: Some("myfs0"), pci_segment: 15 }
[2023-10-25T11:35:44.202Z] cloud-hypervisor: 67.239434s: <vmm> ERROR:virtio-devices/src/vhost_user/vu_common_ctrl.rs:410 -- Failed connecting the backend after trying for 1 minute: VhostUserProtocol(SocketConnect(Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }))
[2023-10-25T11:35:44.202Z] cloud-hypervisor: 67.239491s: <vmm> ERROR:vmm/src/lib.rs:1160 -- Error when adding new fs to the VM: DeviceManager(CreateVirtioFs(VhostUserConnect))

[2023-10-25T11:35:44.210Z] failures:
[2023-10-25T11:35:44.210Z]     common_parallel::test_vfio
[2023-10-25T11:35:44.210Z]     common_parallel::test_virtio_fs
[2023-10-25T11:35:44.210Z]     common_parallel::test_virtio_fs_hotplug
[2023-10-25T11:35:44.210Z]     common_parallel::test_virtio_fs_multi_segment
[2023-10-25T11:35:44.210Z]     common_parallel::test_virtio_fs_multi_segment_hotplug

In previous attempts virtiofsd was increased to 1.4.0. In this PR I considered latest version 1.8.0 then I see below tests are passing in my local machine.

[2023-10-27T14:31:46Z INFO  virtiofsd] Client connected, servicing requests
[2023-10-27T14:31:53Z INFO  virtiofsd] Client connected, servicing requests
[2023-10-27T14:32:19Z INFO  virtiofsd] Client disconnected, shutting down
test common_parallel::test_virtio_fs ... ok
test common_parallel::test_virtio_fs_hotplug has been running for over 60 seconds
test common_parallel::test_virtio_fs_multi_segment_hotplug has been running for over 60 seconds
[2023-10-27T14:32:31Z INFO  virtiofsd] Client disconnected, shutting down
[2023-10-27T14:32:38Z INFO  virtiofsd] Client disconnected, shutting down
[2023-10-27T14:32:41Z INFO  virtiofsd] Waiting for vhost-user socket connection...
[2023-10-27T14:32:48Z INFO  virtiofsd] Waiting for vhost-user socket connection...
[2023-10-27T14:33:01Z INFO  virtiofsd] Client connected, servicing requests
[2023-10-27T14:33:08Z INFO  virtiofsd] Client connected, servicing requests
[2023-10-27T14:33:12Z INFO  virtiofsd] Client disconnected, shutting down
test common_parallel::test_virtio_fs_hotplug ... ok
[2023-10-27T14:33:19Z INFO  virtiofsd] Client disconnected, shutting down
test common_parallel::test_virtio_fs_multi_segment_hotplug ... ok

[2023-10-27T14:35:03Z INFO  virtiofsd] Waiting for vhost-user socket connection...
[2023-10-27T14:35:04Z INFO  virtiofsd] Waiting for vhost-user socket connection...
[2023-10-27T14:35:14Z INFO  virtiofsd] Client connected, servicing requests
[2023-10-27T14:35:20Z INFO  virtiofsd] Client connected, servicing requests
[2023-10-27T14:35:56Z INFO  virtiofsd] Client disconnected, shutting down
test common_parallel::test_virtio_fs_multi_segment ... ok
test common_parallel::test_virtio_fs_multi_segment_hotplug has been running for over 60 seconds
[2023-10-27T14:36:05Z INFO  virtiofsd] Client disconnected, shutting down
[2023-10-27T14:36:15Z INFO  virtiofsd] Waiting for vhost-user socket connection...
[2023-10-27T14:36:35Z INFO  virtiofsd] Client connected, servicing requests
[2023-10-27T14:36:46Z INFO  virtiofsd] Client disconnected, shutting down
test common_parallel::test_virtio_fs_multi_segment_hotplug ... ok

michael2012z · 2023-10-28T08:24:50Z

Regarding the build failure on AArch64, it was also seen in #5487.

There is an issue rust-lang/rust#89626 in rust-lang community tracking the problem. But seems it has not been resolved. Some workarounds were mentioned among the discussion.

rveerama1 · 2023-10-29T17:21:30Z

Regarding the build failure on AArch64, it was also seen in #5487.

Yes, I noticed it.

There is an issue rust-lang/rust#89626 in rust-lang community tracking the problem. But seems it has not been resolved. Some workarounds were mentioned among the discussion.

Let's see if we can find a good workaround.

michael2012z · 2023-10-31T14:16:15Z

@rveerama1, for the AArch64 build error, can you try updating the RUSTFLAGS like this:

diff --git a/scripts/dev_cli.sh b/scripts/dev_cli.sh
index 73b31850..ee9cc48a 100755
--- a/scripts/dev_cli.sh
+++ b/scripts/dev_cli.sh
@@ -286,8 +286,7 @@ cmd_build() {
     rustflags="$RUSTFLAGS"
     target_cc=""
     if [ "$(uname -m)" = "aarch64" ] && [ "$libc" = "musl" ]; then
-        rustflags="$rustflags -C link-arg=-lgcc -C link_arg=-specs -C link_arg=/usr/lib/aarch64-linux-musl/musl-gcc.specs"
-        target_cc="musl-gcc"
+        rustflags="$rustflags -C link-args=-Wl,-Bstatic -C link-args=-lc"
     fi

     $DOCKER_RUNTIME run \
@@ -400,8 +399,7 @@ cmd_tests() {
     rustflags="$RUSTFLAGS"
     target_cc=""
     if [ "$(uname -m)" = "aarch64" ] && [ "$libc" = "musl" ]; then
-        rustflags="$rustflags -C link-arg=-lgcc -C link_arg=-specs -C link_arg=/usr/lib/aarch64-linux-musl/musl-gcc.specs"
-        target_cc="musl-gcc"
+        rustflags="$rustflags -C link-args=-Wl,-Bstatic -C link-args=-lc"
     fi

     if [[ "$unit" = true ]]; then

I can reproduce the error, the change works.

rveerama1 · 2023-10-31T14:24:45Z

@rveerama1, for the AArch64 build error, can you try updating the RUSTFLAGS like this:

Done, Thank you.

rveerama1 · 2023-10-31T16:20:54Z

some update
1)

failures:

---- common_parallel::test_vfio stdout ----

==== Start cloud-hypervisor command-line ====
"target/x86_64-unknown-linux-gnu/release/cloud-hypervisor" "--cpus" "boot=4" "--memory" "size=2G,hugepages=on,shared=on" "--kernel" "/root/workloads/vmlinux" "--disk" "path=/tmp/chZLE0Eu/osdisk.img" "path=/tmp/chZLE0Eu/cloudinit" "path=/root/workloads/vfio.img" "path=/root/workloads/blk.img,iommu=on" "--cmdline" "root=/dev/vda1 console=hvc0 rw systemd.journald.forward_to_console=1 kvm-intel.nested=1 vfio_iommu_type1.allow_unsafe_interrupts" "--net" "tap=vfio-tap0,mac=12:34:56:78:90:00" "tap=vfio-tap1,mac=de:ad:be:ef:12:00,iommu=on" "tap=vfio-tap2,mac=de:ad:be:ef:34:00,iommu=on" "tap=vfio-tap3,mac=de:ad:be:ef:56:00,iommu=on" "-v"
==== End cloud-hypervisor command-line ====

==== Start ssh command output (FAILED) ====
command="grep -c VFIOTAG /proc/cmdline"
auth="PasswordAuth {
    username: "cloud",
    password: "cloud123",
}"
ip="172.18.0.3"
output=""
error="Connection(Os { code: 113, kind: HostUnreachable, message: "No route to host" })"
==== End ssh command outout ====

thread 'common_parallel::test_vfio' panicked at 'called `Result::unwrap()` on an `Err` value: Connection(Os { code: 113, kind: HostUnreachable, message: "No route to host" })', tests/integration.rs:4318:22

test common_parallel::test_vfio ... FAILED , test was unable to ssh into second level (l2) VM. Reason could be second vm didn't boot.

Some guess from @likebreath , "I think the reason why the test_vfio was failing is Cloud Hypervisor binary compiled on new container image Ubuntu 22.04 has different dynamic binaries from what's provided from Ubuntu 20.04 (the focal guest image)."

I modified test_vfio to use Jammy image (Ubuntu 22.04) and tests are passing.

@michael2012z provided a fix for compilation error related to arm workers and it is working and proceeded to integration tests.

Now some tests are failing

arm worker

[2023-10-31T15:20:22.459Z] failures:
[2023-10-31T15:20:22.459Z] 
[2023-10-31T15:20:22.459Z] ---- common_parallel::test_vfio_user stdout ----
[2023-10-31T15:20:22.459Z] thread 'common_parallel::test_vfio_user' panicked at 'assertion failed: exec_host_command_status(\"/usr/local/bin/spdk-nvme/rpc.py nvmf_create_transport -t VFIOUSER\").success()', tests/integration.rs:6737:9

live_migration::live_migration_sequential::test_live_migration_ovs_dpdk failed just like previous attempts. I am looking into this now.

[2023-10-31T15:40:32.680Z] ==== End 'ovs_vm' stderr ====

[2023-10-31T15:40:32.680Z] thread 'live_migration::live_migration_sequential::test_live_migration_ovs_dpdk' panicked at 'Test failed: Error occurred during live-migration', tests/integration.rs:8777:9

windows worker

[2023-10-31T15:29:31.914Z] failures:
[2023-10-31T15:29:31.914Z]     windows::test_windows_guest_disk_hotplug
[2023-10-31T15:29:31.914Z]     windows::test_windows_guest_disk_hotplug_multi

rveerama1 · 2023-11-09T12:18:44Z

No new changes, just rebased against main.

likebreath · 2023-11-09T16:19:37Z

@rveerama1 Index branching will run the CI from time to time even if the PR is not updated and as a draft, and it will waste resources. Let's keep the PR closed unless you want to run the CI. Please reopen as needed. Thank you.

likebreath · 2023-11-09T16:20:22Z

Of course, you will still have the CI log history as a reference for debugging offline: https://cloud-hypervisor-jenkins.westus.cloudapp.azure.com/blue/organizations/jenkins/cloud-hypervisor/activity/?branch=PR-5877

rveerama1 · 2023-11-10T12:35:05Z

@rveerama1 Index branching will run the CI from time to time even if the PR is not updated and as a draft, and it will waste resources. Let's keep the PR closed unless you want to run the CI. Please reopen as needed. Thank you.

Noticed many changes related to integrations tests recently. Thought of checking, will that introduce new issues or solve something on all workers. Anyway I am looking into live_migration::live_migration_sequential::test_live_migration_ovs_dpdk issue so far, no major update towards it. It can be closed for time being.

rveerama1 · 2023-11-15T13:08:10Z

Need some help to investigating it further.

So far live_migration::live_migration_sequential::test_live_migration_ovs_dpdk test has been stuck while running on Ubuntu 22.04 after below steps

cloud-hypervisor: 19.432839s: <vmm> INFO:arch/src/x86_64/mod.rs:579 -- Running under nested virtualisation. Hypervisor string: KVMKVMKVM
cloud-hypervisor: 19.432909s: <vmm> INFO:arch/src/x86_64/mod.rs:585 -- Generating guest CPUID for with physical address size: 40

        self.vu
            .set_log_base(0, Some(log))
            .map_err(Error::VhostUserSetLogBase)?;

it never returned from there.
set_log_base belongs to https://docs.rs/crate/vhost/0.8.1/source/src/vhost_user/master.rs

Don't know exactly what cloud be reason on Ubuntu22.04 it stuck over there.

some sample logs from Ubuntu 20.04

cloud-hypervisor: 85.599615s: <vmm> INFO:arch/src/x86_64/mod.rs:579 -- Running under nested virtualisation. Hypervisor string: KVMKVMKVM
cloud-hypervisor: 85.599689s: <vmm> INFO:arch/src/x86_64/mod.rs:585 -- Generating guest CPUID for with physical address size: 40
cloud-hypervisor: 86.828570s: <vmm> INFO:vmm/src/lib.rs:1743 -- Dirty memory migration 0 of 5
cloud-hypervisor: 86.829669s: <vmm> INFO:vmm/src/memory_manager.rs:2679 -- Dirty Memory Range Table:
cloud-hypervisor: 86.829698s: <vmm> INFO:vmm/src/memory_manager.rs:2681 -- GPA: 2003000 size: 4 (KiB)
cloud-hypervisor: 86.829736s: <vmm> INFO:vmm/src/memory_manager.rs:2681 -- GPA: 2007000 size: 4 (KiB)
cloud-hypervisor: 86.829782s: <vmm> INFO:vmm/src/memory_manager.rs:2681 -- GPA: 200f000 size: 8 (KiB)
cloud-hypervisor: 86.829848s: <vmm> INFO:vmm/src/memory_manager.rs:2681 -- GPA: 2014000 size: 8 (KiB)
cloud-hypervisor: 86.829902s: <vmm> INFO:vmm/src/memory_manager.rs:2681 -- GPA: 2035000 size: 4 (KiB)

It does proceed further with dirty memory migration.

Any help/suggestion/insights @likebreath @rbradford @sboeuf ?

rveerama1 · 2023-11-16T10:04:39Z

Marking as WIP doesn't trigger CI.

likebreath · 2023-11-16T22:45:20Z

finally it stuck at here https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/virtio-devices/src/vhost_user/vu_common_ctrl.rs#L500
        self.vu
            .set_log_base(0, Some(log))
            .map_err(Error::VhostUserSetLogBase)?; 
it never returned from there. set_log_base belongs to https://docs.rs/crate/vhost/0.8.1/source/src/vhost_user/master.rs

@rveerama1 Good progress. It is now clear that the problem is from migrating vhost_user device (that is using a ovs-dpdk backend) while sending request SET_LOG_BASE to the backend. I'd suggest you locate precisely where the hang comes from in the vhost crate and open an issue from the vhost crate repository to get some inputs [1].

Note that we recently upgraded vhost crate version. So please rebase before further debugging.

[1] https://github.com/rust-vmm/vhost

rveerama1 · 2023-11-17T13:35:31Z

Note that we recently upgraded vhost crate version. So please rebase before further debugging.

Ok, I will check and update.

rveerama1 · 2023-12-13T12:17:43Z

~~Also this error~~

[2023-12-12T12:53:19.501Z] ERROR: Build data file '/root/workloads/spdk/build/libvfio-user/build-release/meson-private/build.dat' references functions or classes that don't exist. This probably means that it was generated with an old version of meson. Consider reconfiguring the directory with "meson setup --reconfigure".
[2023-12-12T12:53:19.501Z] make[1]: *** [Makefile:21: build] Error 1
[2023-12-12T12:53:19.501Z] make: *** [/root/workloads/spdk/mk/spdk.subdirs.mk:16: vfiouserbuild] Error 2
[2023-12-12T12:53:19.501Z] make: *** Waiting for unfinished jobs....

rveerama1 · 2023-12-13T13:02:03Z

ARM worker fixed as well.

rveerama1 · 2023-12-13T14:17:52Z

On windows worker two tests were failing
[2023-11-30T13:29:03.480Z] failures:
[2023-11-30T13:29:03.480Z]     windows::test_windows_guest_disk_hotplug
[2023-11-30T13:29:03.480Z]     windows::test_windows_guest_disk_hotplug_multi
@likebreath @liuw can you request someone to take a look on windows worker side.
@weltling Would you please help take a look?

Disabled those two tests. Now CI looks fine.

tests/integration.rs

likebreath

We will need to run change more times to see how stable our tests are on the migrated docker container. Also, we will need to test bare-metal workers.

Since we are cutting a new release tomorrow. Let's do these tests after the release. Marking this PR as DNM to avoid unnecessary CI runs before that.

rveerama1 · 2023-12-14T10:03:47Z

it seems spdk needs to rebuild every time?
it started throwing error again on ARM side

[2023-12-14T09:52:16.961Z] cloud-hypervisor: 6.856043s: <vmm> INFO:vmm/src/lib.rs:1969 -- API request event: VmInfo(Sender { .. })
[2023-12-14T09:52:20.395Z] /usr/local/bin/spdk-nvme/nvmf_tgt: error while loading shared libraries: libjson-c.so.4: cannot open shared object file: No such file or directory

[2023-12-14T09:57:57.622Z] ---stdout---
[2023-12-14T09:57:57.622Z] Error while connecting to /var/tmp/spdk.sock
[2023-12-14T09:57:57.622Z] Is SPDK application running?
[2023-12-14T09:57:57.622Z] Error details: Invalid or non-existing address: '/var/tmp/spdk.sock'
[2023-12-14T09:57:57.622Z] 
[2023-12-14T09:57:57.622Z] ---stderr--- 
[2023-12-14T09:57:57.622Z] 
[2023-12-14T09:57:57.622Z] ==== End 'exec_host_command' failed ====
[2023-12-14T09:57:57.622Z] thread 'common_parallel::test_vfio_user' panicked at 'assertion failed: exec_host_command_status(\"/usr/local/bin/spdk-nvme/rpc.py nvmf_create_transport -t VFIOUSER\").success()',

It was picking right shared libraries of libjson-c.so.5 and initialization of nvmf_tgt. But now container picking again old libs.
Seems something wrong with how container works.

rveerama1 · 2023-12-14T11:53:29Z

So for clean SPDK builds, it works properly.
If we build once and copy only binaries like in the scripts then test_vfio_user test goes flaky.

likebreath · 2023-12-14T16:00:56Z

DNM does not stopthe CI pipeline from building, only WIP or RFC does.

The following tests have been temporarily disabled: 1. Live upgrade/migration test with ovs-dpdk (cloud-hypervisor#5532); 2. Disk hotplug tests on windows guests (cloud-hypervisor#6037); This patch has been tested with PR cloud-hypervisor#6048. Signed-off-by: Ravi kumar Veeramally <ravikumar.veeramally@intel.com> Signed-off-by: Michael Zhao <michael.zhao@arm.com> Tested-by: Bo Chen <chen.bo@intel.com>

likebreath · 2023-12-20T19:26:34Z

This patch has been tested with #6048. Details: #6048 (comment). I updated the container tag and added details about the lists being disabled in the commit message.

I think we are ready to land this PR. Thank you for the good work @rveerama1.

rveerama1 requested a review from a team as a code owner October 25, 2023 11:10

rveerama1 marked this pull request as draft October 25, 2023 11:10

likebreath mentioned this pull request Oct 25, 2023

Migrate docker container image from ubuntu 20.04 to 22.04 #5878

Closed

rveerama1 force-pushed the issue_5532 branch from c9f7db7 to 8f761e0 Compare October 27, 2023 15:49

rveerama1 force-pushed the issue_5532 branch from 8f761e0 to a190102 Compare October 31, 2023 12:10

rveerama1 closed this Oct 31, 2023

rveerama1 reopened this Oct 31, 2023

rveerama1 force-pushed the issue_5532 branch from a190102 to 5e4848c Compare October 31, 2023 14:24

rveerama1 force-pushed the issue_5532 branch from 5e4848c to c71f999 Compare October 31, 2023 15:06

rveerama1 force-pushed the issue_5532 branch from c71f999 to c5fa68d Compare November 9, 2023 12:18

likebreath closed this Nov 9, 2023

rveerama1 changed the title ~~Migrate docker container from ubuntu 20.04 to 22.04~~ [WIP] Migrate docker container from ubuntu 20.04 to 22.04 Nov 16, 2023

rveerama1 reopened this Nov 16, 2023

rveerama1 changed the title ~~[WIP] Migrate docker container from ubuntu 20.04 to 22.04~~ Migrate docker container from ubuntu 20.04 to 22.04 Dec 13, 2023

rveerama1 force-pushed the issue_5532 branch from d533d47 to 42247d1 Compare December 13, 2023 12:34

rveerama1 force-pushed the issue_5532 branch from 42247d1 to 498b78a Compare December 13, 2023 13:23

rveerama1 force-pushed the issue_5532 branch from 498b78a to 2ecf516 Compare December 13, 2023 14:18

rveerama1 marked this pull request as ready for review December 13, 2023 15:08

rbradford reviewed Dec 13, 2023

View reviewed changes

tests/integration.rs Outdated Show resolved Hide resolved

tests/integration.rs Outdated Show resolved Hide resolved

likebreath reviewed Dec 13, 2023

View reviewed changes

likebreath changed the title ~~Migrate docker container from ubuntu 20.04 to 22.04~~ [DNM] Migrate docker container from ubuntu 20.04 to 22.04 Dec 13, 2023

likebreath mentioned this pull request Dec 13, 2023

Fail to build spdk on aarch64 #6030

Closed

rveerama1 mentioned this pull request Dec 14, 2023

Windows guest disk hotplug tests fails on migrated CI Ubuntu 22.04 container image #6037

Open

rveerama1 force-pushed the issue_5532 branch from 2ecf516 to 0976355 Compare December 14, 2023 08:59

rveerama1 force-pushed the issue_5532 branch 2 times, most recently from 7ee76b4 to 0976355 Compare December 14, 2023 12:08

likebreath changed the title ~~[DNM] Migrate docker container from ubuntu 20.04 to 22.04~~ [WIP] Migrate docker container from ubuntu 20.04 to 22.04 Dec 14, 2023

likebreath force-pushed the issue_5532 branch from aecfa5e to 2d90980 Compare December 14, 2023 16:21

likebreath mentioned this pull request Dec 18, 2023

Test PR #5877 on bare-metal workers #6048

Closed

likebreath force-pushed the issue_5532 branch from 2d90980 to a824f5b Compare December 20, 2023 19:19

likebreath changed the title ~~[WIP] Migrate docker container from ubuntu 20.04 to 22.04~~ Migrate docker container from ubuntu 20.04 to 22.04 Dec 20, 2023

likebreath force-pushed the issue_5532 branch from a824f5b to de5492e Compare December 20, 2023 19:23

likebreath approved these changes Dec 20, 2023

View reviewed changes

likebreath enabled auto-merge (rebase) December 20, 2023 19:53

likebreath merged commit 24f384d into cloud-hypervisor:main Dec 20, 2023
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate docker container from ubuntu 20.04 to 22.04 #5877

Migrate docker container from ubuntu 20.04 to 22.04 #5877

rveerama1 commented Oct 25, 2023

likebreath commented Oct 25, 2023

likebreath commented Oct 25, 2023

rveerama1 commented Oct 26, 2023 •

edited

rveerama1 commented Oct 27, 2023 •

edited

michael2012z commented Oct 28, 2023

rveerama1 commented Oct 29, 2023

michael2012z commented Oct 31, 2023

rveerama1 commented Oct 31, 2023

rveerama1 commented Oct 31, 2023

rveerama1 commented Nov 9, 2023

likebreath commented Nov 9, 2023

likebreath commented Nov 9, 2023

rveerama1 commented Nov 10, 2023

rveerama1 commented Nov 15, 2023

rveerama1 commented Nov 16, 2023

likebreath commented Nov 16, 2023

rveerama1 commented Nov 17, 2023

rveerama1 commented Dec 13, 2023 •

edited

rveerama1 commented Dec 13, 2023

rveerama1 commented Dec 13, 2023

likebreath left a comment

rveerama1 commented Dec 14, 2023

rveerama1 commented Dec 14, 2023

likebreath commented Dec 14, 2023

likebreath commented Dec 20, 2023

Migrate docker container from ubuntu 20.04 to 22.04 #5877

Migrate docker container from ubuntu 20.04 to 22.04 #5877

Conversation

rveerama1 commented Oct 25, 2023

likebreath commented Oct 25, 2023

likebreath commented Oct 25, 2023

rveerama1 commented Oct 26, 2023 • edited

rveerama1 commented Oct 27, 2023 • edited

michael2012z commented Oct 28, 2023

rveerama1 commented Oct 29, 2023

michael2012z commented Oct 31, 2023

rveerama1 commented Oct 31, 2023

rveerama1 commented Oct 31, 2023

rveerama1 commented Nov 9, 2023

likebreath commented Nov 9, 2023

likebreath commented Nov 9, 2023

rveerama1 commented Nov 10, 2023

rveerama1 commented Nov 15, 2023

rveerama1 commented Nov 16, 2023

likebreath commented Nov 16, 2023

rveerama1 commented Nov 17, 2023

rveerama1 commented Dec 13, 2023 • edited

rveerama1 commented Dec 13, 2023

rveerama1 commented Dec 13, 2023

likebreath left a comment

Choose a reason for hiding this comment

rveerama1 commented Dec 14, 2023

rveerama1 commented Dec 14, 2023

likebreath commented Dec 14, 2023

likebreath commented Dec 20, 2023

rveerama1 commented Oct 26, 2023 •

edited

rveerama1 commented Oct 27, 2023 •

edited

rveerama1 commented Dec 13, 2023 •

edited