Update to OpenZFS 2.2.2 #3779

andrewd-zededa · 2024-02-27T20:58:43Z

Kernel update - [amd64-generic, amd64-generic, amd64-rt, arm64-nvidia, arm64-generic, arm64-generic]

eve-kernel-amd64-v5.10.186-generic
4d7866684f79: Update to OpenZFS 2.2.2

eve-kernel-amd64-v6.1.38-generic
820be3b1c356: Update to OpenZFS 2.2.2

eve-kernel-amd64-v6.1.38-rt
7148ced52fc1: Update to OpenZFS 2.2.2

eve-kernel-arm64-v5.10.104-nvidia
ab0df9cfc431: Update to OpenZFS 2.2.2

eve-kernel-arm64-v5.10.186-generic
438959506e33: Update to OpenZFS 2.2.2

eve-kernel-arm64-v6.1.38-generic
50c6e530bbd2: Update to OpenZFS 2.2.2

Remove modifications to now unused kernel module parameters.
Don't apply patches as they're now included in upstream.
Remove now unused patches under dom0-ztools
Patch go-libzfs for OpenZFS 2.2.2 support

eriknordmark · 2024-02-27T21:02:19Z

We need this in the 11.0-stable branch as well.

codecov · 2024-02-27T21:09:05Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 19.69%. Comparing base (a50198a) to head (0ce6a55).
Report is 10 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3779   +/-   ##
=======================================
  Coverage   19.69%   19.69%           
=======================================
  Files         235      235           
  Lines       51757    51757           
=======================================
+ Hits        10192    10193    +1     
+ Misses      40825    40823    -2     
- Partials      740      741    +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

andrewd-zededa · 2024-02-27T23:38:57Z

We need this in the 11.0-stable branch as well.

It's on the way

andrewd-zededa · 2024-02-27T23:52:58Z

@eriknordmark Here is the same PR for 11.0-stable #3780

pkg/dom0-ztools/rootfs/etc/init.d/000-mod-params

eriknordmark

Kick off tests

rouming · 2024-02-28T07:53:32Z

@eriknordmark do we want to merge this to stable branches as well?

Update: I found your comment, where you say we need this to backport to the 11.0-stable. Only there, correct?

rene · 2024-02-28T10:29:20Z

pkg/dom0-ztools/Dockerfile

 ENV ZFS_COMMIT=zfs-${ZFS_VERSION}
 ENV ZFS_REPO=https://github.com/openzfs/zfs
-ENV ZFS_PATCH_DIR=/patches-zfs-"${ZFS_VERSION}"

 WORKDIR /tmp/zfs

 ADD ${ZFS_REPO}/tarball/${ZFS_COMMIT}/ zfs.tgz
 RUN tar -zxvf zfs.tgz  --strip-components=1 && \


@andrewd-zededa since you are touching here I would like to propose an improvement apart from your required changes: Eliminate the additional step of extracting the tarball.... we can just simply fetch directly from github using the ADD command, like this:

diff --git a/pkg/dom0-ztools/Dockerfile b/pkg/dom0-ztools/Dockerfile index dcd7c86a6..3ba4a396c 100644 --- a/pkg/dom0-ztools/Dockerfile +++ b/pkg/dom0-ztools/Dockerfile @@ -10,14 +10,12 @@ COPY /patches / # * ZFS on Linux ENV ZFS_VERSION=2.1.2 ENV ZFS_COMMIT=zfs-${ZFS_VERSION} -ENV ZFS_REPO=https://github.com/openzfs/zfs +ENV ZFS_REPO=https://github.com/openzfs/zfs.git ENV ZFS_PATCH_DIR=/patches-zfs-"${ZFS_VERSION}" WORKDIR /tmp/zfs +ADD ${ZFS_REPO}#${ZFS_COMMIT} /tmp/zfs -ADD ${ZFS_REPO}/tarball/${ZFS_COMMIT}/ zfs.tgz -RUN tar -zxvf zfs.tgz --strip-components=1 && \ - rm zfs.tgz

Sure I'll make the change.

rene · 2024-02-28T10:34:07Z

pkg/pillar/go-module-patches/001-go-libzfs-zfs-2-2-2-support.patch

+ 		vdevs.ScanStat.ToExamine = uint64(ps.pss_to_examine)
+ 		vdevs.ScanStat.Examined = uint64(ps.pss_examined)
+-		vdevs.ScanStat.ToProcess = uint64(ps.pss_to_process)
+		//vdevs.ScanStat.ToProcess = uint64(ps.pss_to_process)


If you don't need this line, just remove it......

I'll fix this in my go-libzfs fork.

Fixed in fork

rene · 2024-02-28T10:45:02Z

pkg/pillar/Dockerfile

@@ -92,6 +92,12 @@ ENV GOARCH=${TARGETARCH}
 ENV CC=${CROSS_COMPILE_ENV}gcc
 ARG GOPKGVERSION

+# Temporary gomodule patches until accepted upstream


We've never patched go modules directly like this... IMO would be better to push this modified version to a fork on GH (you did these changes, right? So you can push them to a forked repo under your account) and use the fork as the GO module (changing at the go.mod). Once the changes become upstream, you can update the go.mod to fetch from the official repo again...

Yes I authored this patch. Sounds good, I'll make the changes to importing a module instead.

eriknordmark · 2024-02-28T13:23:00Z

@eriknordmark do we want to merge this to stable branches as well?

Update: I found your comment, where you say we need this to backport to the 11.0-stable. Only there, correct?

Yes, because with the old kernel (which lived in eve/pkg/kernel) we had the old ZFS patches to get the same behavior.

rene · 2024-02-29T11:00:52Z

kernel-commits.mk

-KERNEL_COMMIT_amd64_v6.1.38_rt = 49dba60a59d9
-KERNEL_COMMIT_arm64_v5.10.104_nvidia = c21d5a4dd498
-KERNEL_COMMIT_arm64_v5.10.186_generic = 6a861e0433a6
-KERNEL_COMMIT_arm64_v6.1.38_generic = 92466d23a9bf


@andrewd-zededa , please, move the kernel update to a dedicated commit, so it helps with traceability (our kernel updates are all in dedicated commits) and separates logically the changes of your PR....

Split the kernel change out to its own commit. I can squash the other two commits down to one if needed.

andrewd-zededa · 2024-03-06T04:34:13Z

@rene @christoph-zededa @zedi-pramodh

Please review my latest changes as time allows. If this looks good I will copy them to the 11.0-stable PR as well.

rene · 2024-03-06T16:10:37Z

@rene @christoph-zededa @zedi-pramodh

Please review my latest changes as time allows. If this looks good I will copy them to the 11.0-stable PR as well.

@andrewd-zededa , LGTM, I have just some final comments:

You need to rebase on top of master (Yetus is complaining)
The PR Kernel update - [amd64-generic, amd64-generic, amd64-rt, amd64-generi… #3793 contains newer kernel updates which covers your changes, so you can drop the commit which updates the kernel. This is also one of the reasons to keep kernel updates into a single commit, now you can very easily drop your commit and we are done.
Once you drop the kernel updates, update the description of this PR mentioning the dependency of PR 3793

andrewd-zededa · 2024-03-07T02:59:30Z

@rene I rebased to fix the first yetus issue. The next yetus issue "could not import github.com/andrewd-zededa/go-libzfs" is not new in my PR, here's an example of it seen with the old bicomsystems version of the module.

cmd/zfsmanager/handlediskconfig.go:12:9: could not import github.com/bicomsystems/go-libzfs (-: # github.com/bicomsystems/go-libzfs
vendor/github.com/bicomsystems/go-libzfs/common.go:16:10: fatal error: 'libzfs.h' file not found
#include <libzfs.h>
         ^~~~~~~~~~
1 error generated.) (typecheck)
        libzfs "github.com/bicomsystems/go-libzfs"

Since 3793 has not been merged yet can we instead merge this one first as a complete pr?

rene · 2024-03-07T11:32:06Z

@rene I rebased to fix the first yetus issue. The next yetus issue "could not import github.com/andrewd-zededa/go-libzfs" is not new in my PR, here's an example of it seen with the old bicomsystems version of the module.
cmd/zfsmanager/handlediskconfig.go:12:9: could not import github.com/bicomsystems/go-libzfs (-: # github.com/bicomsystems/go-libzfs
vendor/github.com/bicomsystems/go-libzfs/common.go:16:10: fatal error: 'libzfs.h' file not found
#include <libzfs.h>
         ^~~~~~~~~~
1 error generated.) (typecheck)
        libzfs "github.com/bicomsystems/go-libzfs"
Since 3793 has not been merged yet can we instead merge this one first as a complete pr?

Thanks @andrewd-zededa , I checked this yetus error, that's happening due to the lack of the libzfslinux-dev package inside the yetus container. We can ignore it in your PR, but we'll need to think in a proper fix for that...

As you probably saw, the PR #3793 was merged. So I think you just need to drop the kernel update commit and we are good to go...

rene · 2024-03-07T14:19:42Z

@andrewd-zededa , I've started a PR to fix Yetus issue: #3798 , feel free to review and comment on it... it's still a draft because installing the libzfs is now leading to another Yetus errors....

Edit:
Building the libzfs from sources (with the version expected by pillar) it works. Feel free to check with your changes as well.

Remove now unused patches which are in upstream. Remove module param modifications which no longer exist. Signed-off-by: Andrew Durbin <andrewd@zededa.com>

Migrate go-libzfs to a fork including 2.2.2 support. Signed-off-by: Andrew Durbin <andrewd@zededa.com>

andrewd-zededa · 2024-03-07T17:00:23Z

Ok @rene, dropped the kernel commit.

rene

LGTM

eriknordmark · 2024-03-08T08:22:47Z

See comment in #3780

eriknordmark · 2024-03-11T14:04:01Z

pkg/pillar/Dockerfile

@@ -10,7 +10,7 @@ ARG BUILD_PKGS_BASE="git gcc linux-headers libc-dev make linux-pam-dev m4 findut
 # we use the same image in several places
 ARG EVE_ALPINE_IMAGE=lfedge/eve-alpine:9fb9b9cbf7d90066a70e4704d04a6fe248ff52bb

-FROM lfedge/eve-dom0-ztools:417d4ff6a57d2317c9e65166274b0ea6f6da16e2 as zfs
+FROM lfedge/eve-dom0-ztools:81a7af28b32c7558e1c4622e60a9e3197075decd as zfs


Does this not pull in the correct version with the include files etc?

The unit tests fail consistently. How can we fix that?
Any ideas @rene ?

Can it be that it is like when we update eve-alpine? First the PR for eve-dom0-ztools has to be merged and only then it can be used in other Dockerfiles?

eriknordmark

Need to get the unit tests to build and pass.

andrewd-zededa · 2024-03-11T19:58:20Z

@erik @deitch @rene Latest commit adds zfs build to build-tools/src/scripts/Dockerfile so that 'make test' runs.

Now libzfs headers are available for go-libzfs cgo build. Signed-off-by: Andrew Durbin <andrewd@zededa.com>

eriknordmark

It seems like with #3804 we can avoid having to build this and avoid keeping the version of zfs packages in the build container in synch with the kernel.

andrewd-zededa · 2024-03-12T05:00:47Z

@eriknordmark should I back out my latest commit and then this can merge after #3804 merges?

deitch

Does this move to OpenZFS 2.2.2 eliminate those custom patch files? If so, that is a lovely thing.

Separately, why are we depending on a custom fork of go-libzfs which is in a personal repo? I refer to this and about 10 other places.

eriknordmark · 2024-03-12T07:12:50Z

@eriknordmark should I back out my latest commit and then this can merge after #3804 merges?

I think that makes sense unless @rene thinks that is a bad idea.

eriknordmark · 2024-03-12T07:23:44Z

Does this move to OpenZFS 2.2.2 eliminate those custom patch files? If so, that is a lovely thing.

[Answering for Andrew since it is late for him]
Yes

Separately, why are we depending on a custom fork of go-libzfs which is in a personal repo? I refer to this and about 10 other places.

Some fixes were needed for 2.2.2. See bicomsystems/go-libzfs@master...andrewd-zededa:go-libzfs:andrewd-zfs-2.2.2

Plan is to submit a PR for that upstream. Don't know if @andrewd-zededa has already started the upstreaming.

deitch · 2024-03-12T08:02:07Z

Plan is to submit a PR for that upstream

I think they usually are pretty responsive. We should link to that PR before we start to depend on a fork. Is the PR there? If not, can we see the diff here?

christoph-zededa · 2024-03-12T12:21:01Z

@eriknordmark should I back out my latest commit and then this can merge after #3804 merges?

I think that makes sense unless @rene thinks that is a bad idea.

To me it still makes sense to put openzfs into the build-tools-container; I am not sure about others but I use it every time for development of pillar.

Also I would suggest to first merge this PR and only then #3804 because then we already have the new version of the pillar container pushed to the registry and avoid splitting this PR into two PRs.

eriknordmark

If the team wants to include the zfs packages in the build container I can live with that. But it means an additional maintenance burden to update that when there is a new version of ZFS.

@andrewd-zededa can you start the discussion (maybe in the form of a github issue) for the upstream zfs repo and see if they will take your changes?

andrewd-zededa · 2024-03-12T14:52:17Z

@eriknordmark yes I'll start the upstream discussion. The correct diff link of my fork which my PR uses is here: bicomsystems/go-libzfs@v0.4.0...andrewd-zededa:go-libzfs:andrewd-zfs-2.2.2-0.4.0

I originally intended to base my changes off their master branch but it seems its missing one or more changes which are in v0.4.0 (the one eve seems to use currently). So the module version linked in my PR here is a branch off of v0.4.0 at the moment.

deitch · 2024-03-12T14:57:20Z

I originally intended to base my changes off their master branch but it seems its missing one or more changes which are in v0.4.0 (the one eve seems to use currently).

master is missing things in v0.4.0? That is strange.

The change is pretty small (at least diffing to v0.4.0), so should get it in and have them engage.

andrewd-zededa · 2024-03-12T19:00:14Z

I have a starting point with questions on bicomsystems's repo here: bicomsystems/go-libzfs#51.

When I hear back from them i'll move forward.

eriknordmark · 2024-03-12T20:19:28Z

I'll merge this now and we can see how soon we have the support in the upstream.

andrewd-zededa requested review from rouming and eriknordmark as code owners February 27, 2024 20:58

github-actions bot requested a review from christoph-zededa February 27, 2024 20:59

eriknordmark added the stable Should be backported to stable release(s) label Feb 27, 2024

eriknordmark reviewed Feb 28, 2024

View reviewed changes

pkg/dom0-ztools/rootfs/etc/init.d/000-mod-params Show resolved Hide resolved

eriknordmark approved these changes Feb 28, 2024

View reviewed changes

rouming approved these changes Feb 28, 2024

View reviewed changes

rene reviewed Feb 28, 2024

View reviewed changes

rene reviewed Feb 29, 2024

View reviewed changes

andrewd-zededa force-pushed the andrewd-zfs-2.2.2 branch from b382721 to f11b95d Compare March 5, 2024 00:21

andrewd-zededa requested a review from zedi-pramodh as a code owner March 5, 2024 00:21

github-actions bot requested review from eriknordmark and rouming March 5, 2024 00:22

rouming approved these changes Mar 5, 2024

View reviewed changes

christoph-zededa approved these changes Mar 6, 2024

View reviewed changes

andrewd-zededa force-pushed the andrewd-zfs-2.2.2 branch from f11b95d to 8326f97 Compare March 6, 2024 18:38

github-actions bot requested review from christoph-zededa and rouming March 6, 2024 18:39

andrewd-zededa force-pushed the andrewd-zfs-2.2.2 branch from 8326f97 to 86941af Compare March 6, 2024 21:48

andrewd-zededa added 2 commits March 7, 2024 09:59

Update dom0-ztools to OpenZFS 2.2.2

4229ebe

Remove now unused patches which are in upstream. Remove module param modifications which no longer exist. Signed-off-by: Andrew Durbin <andrewd@zededa.com>

Update to OpenZFS 2.2.2

e0ea38d

Migrate go-libzfs to a fork including 2.2.2 support. Signed-off-by: Andrew Durbin <andrewd@zededa.com>

andrewd-zededa force-pushed the andrewd-zfs-2.2.2 branch from 86941af to e0ea38d Compare March 7, 2024 16:59

rene approved these changes Mar 7, 2024

View reviewed changes

andrewd-zededa mentioned this pull request Mar 7, 2024

[11.0-stable] Update to OpenZFS 2.2.2 #3780

Merged

andrewd-zededa mentioned this pull request Mar 8, 2024

yetus: Install dependencies inside Yetus container #3798

Closed

eriknordmark reviewed Mar 11, 2024

View reviewed changes

eriknordmark requested changes Mar 11, 2024

View reviewed changes

andrewd-zededa requested a review from deitch as a code owner March 11, 2024 19:53

github-actions bot requested a review from eriknordmark March 11, 2024 19:55

andrewd-zededa force-pushed the andrewd-zfs-2.2.2 branch from 8317674 to 751d367 Compare March 11, 2024 20:06

Build OpenZFS 2.2.2 in build-tools to fix 'make test'

0ce6a55

Now libzfs headers are available for go-libzfs cgo build. Signed-off-by: Andrew Durbin <andrewd@zededa.com>

andrewd-zededa force-pushed the andrewd-zfs-2.2.2 branch from 751d367 to 0ce6a55 Compare March 11, 2024 20:49

eriknordmark reviewed Mar 11, 2024

View reviewed changes

deitch reviewed Mar 12, 2024

View reviewed changes

eriknordmark approved these changes Mar 12, 2024

View reviewed changes

eriknordmark merged commit 343ec23 into lf-edge:master Mar 12, 2024
36 of 40 checks passed

Update to OpenZFS 2.2.2 #3779

Update to OpenZFS 2.2.2 #3779

Conversation

andrewd-zededa commented Feb 27, 2024

eriknordmark commented Feb 27, 2024

codecov bot commented Feb 27, 2024 • edited Loading

Codecov Report

andrewd-zededa commented Feb 27, 2024

andrewd-zededa commented Feb 27, 2024

eriknordmark left a comment

Choose a reason for hiding this comment

rouming commented Feb 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eriknordmark commented Feb 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewd-zededa commented Mar 6, 2024

rene commented Mar 6, 2024

andrewd-zededa commented Mar 7, 2024

rene commented Mar 7, 2024

rene commented Mar 7, 2024 • edited Loading

andrewd-zededa commented Mar 7, 2024

rene left a comment

Choose a reason for hiding this comment

eriknordmark commented Mar 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eriknordmark left a comment

Choose a reason for hiding this comment

andrewd-zededa commented Mar 11, 2024

eriknordmark left a comment

Choose a reason for hiding this comment

andrewd-zededa commented Mar 12, 2024

deitch left a comment

Choose a reason for hiding this comment

eriknordmark commented Mar 12, 2024

eriknordmark commented Mar 12, 2024

deitch commented Mar 12, 2024

christoph-zededa commented Mar 12, 2024

eriknordmark left a comment

Choose a reason for hiding this comment

andrewd-zededa commented Mar 12, 2024

deitch commented Mar 12, 2024

andrewd-zededa commented Mar 12, 2024

eriknordmark commented Mar 12, 2024

codecov bot commented Feb 27, 2024 •

edited

Loading

rouming commented Feb 28, 2024 •

edited

Loading

rene commented Mar 7, 2024 •

edited

Loading