Added minimal CI + Signing section #503
Conversation
|
Thanks for this, this looks really cool - I'll give it a proper look when I get the chance. One thing that does stand out is that I think "Release" should be "RelWithDebInfo" - and it needs to be saving the PDB files somewhere. As for signing, I think you can generate a dummy key that's usable in test mode. I have to use a USB dongle for my actual key unfortunately, so I can't upload that. |
|
Here we go ! Test-signing the driver is really harder than it look |
|
Okay, after thinking about it, the "test-signing" thing seems way too complicated/dangerous to do from the perspective of an user wanting to test a nightly version (disabling secure boot, putting the computer into a test mode, installing the WDK, adding a trusted CA to the computer) Compared to just... Disabling signature checking |
|
Done ! I'd really like a review on my additions to the README |
Hum... maybe ? @maharmstone, do you want me to take a look at it ? (or do you not want a key in the CI anyway) |
|
Please add ci badge on top README 😉 PS: Windows 11 Pro last update, driver works fine |
|
I literally copied it from another repository, please tell me if I should change anything |
iTrooz
force-pushed
the
master
branch
3 times, most recently
from
8f09ecb
to
76ec7e8
Compare
|
Okay I struggled way more than I should have, but it's done |
|
you might be interested in taking a look at what i did this has a script for generating certs this has some some commands for importing certificates to use for signing |
|
Thanks @andrewc12, looks like it could be useful. |
|
I'll take a look at it when I get time ! |
|
Seems great ! |
|
That part I didn't work on I should also say alot of my inspiration came from the usbip driver documentation This is where they sign their driver if you're interested. |
|
Indeed ! I'm sad all of my work on the README will get scrapped lol |
|
I just tested a script to install a cert as a root cert I found the info about the cert store names here |
|
Why would you want to install it as root in the CI ? |
|
I meant as a root certificate authority |
|
Ohhhhh, right ! Didn't think of that |
|
After reviewing your CI, I found something that worries me |
|
For us this is absolutely fine since it is only used on development devices. |
|
Hum... I understand your point, but this still scares me tbh |
|
(If for any reason someone ever needs what I wrote in the README, which I'm going to erase at least partly : https://gist.github.com/iTrooz/da1a4fd780e3bfc0c10ec63fa55fe447) |
|
Nice |
|
Say, while rewrting the README I realized something The certificate does not need to be installed on the user's computer to be able to install the driver So... Is installing the certificate on the user's machine really that important ? |
|
I'm not sure about with what is happening here but with openzfsonwindows you either need to
|
|
Interesting I guess I will tell users to still install the certificate, in case Microsoft randomly decides to remove this warning and outright not install the driver |
|
By the way @maharmstone, could you explain me how test.exe works ? I'd like to run these tests in the CI, but I can't figure out how to use it. I tried to put the path to a btrfs device as an argument ( |
|
Could this be caused by the test program retuning an error and then github hiding everything else? Maybe try running it once where it's So that it doesn't error out and you still get the test output for example I get something similar running tests finally I feel bad even mentioning it but what about trying E:\ ? |
|
And I feel even worse because.... it works.. |
|
This is kinda funny to me because you're like where I was a week ago for workflows. I kept messing these things up all the time. |
|
Lol Is your workflow working now ? |
|
By the way, some tests fail (400 out of 2500), which is weird, because, well... They shouldn't ? EDIT : some tests fails even on C:/, really weird |
Yeah it is. Which I did by creating a dummy test step at the top of the tests that would only run if everything went well up untill that point It's pretty good right now, but I need to change it so it throws an error if the successful tests is less than the total tests Right now my thoughts are to use PowerShell select-string to grab the n and m from "passed n/m" |
Could you link to this ? I'm not sure I understand
Why not make the process return a non-zero exit code ? |
https://github.com/andrewc12/openzfs/blob/andrew_workflows-11_remainder/.github/workflows/wf.yaml#L120-L228
That's what I meant by throwing an error |
|
Is anything missing for this to get merged ? @maharmstone |
Yes - the time I'll need to have a look at this. I'm not going to merge anything I don't understand myself, and at the moment my knowledge of both CI/CD and GitHub actions is limited. I think it'd likely be a lot quicker if the CI process used Clang in Linux, rather than spinning up a Windows VM and installing MSVC... but I still need to see if Clang works for this, and if so how good it would be. GCC's a non-starter unfortunately, as it doesn't support either SEH or PDBs (though I'm working on that). Also, as far as I can see, when it comes to Windows VMs GitHub only lets you use Windows 10, which isn't all that useful. I'd prefer something that lets you run tests on umpteen different versions of Windows, such as Wine has. It looks like this is possible with GitHub actions, but I'd have to set up and host the VMs etc. myself. |
|
Oh sorry, take the time you need
Note that CI time is free for github repositories (the 2000 minutes per account thing is for private repositories), so while I understand your concerns about speed, I don't think it's this much of a problem (On the other hand, I recently realised that the pbd file size might be a problem because you only have so much storage. You may want to diminish the storage period for artifacts, or move the pdb generation to a separate CI that runs on demand with a workflow_dispatch call)
If you are talking about the self hosted runners, yes (or I think you could use another provider to get the W11/etc.. runners, and make github use them, no experience here) |
|
Thanks ! I had to apply all of these manually because the github UI wouldn't let me automatically add your suggestions, so I hope I didn't miss anything |
|
Nope, I ran it directly on my PC with a btrfs-formatted USB stick
I'm not even sure where to begin to mount a btrfs volume on the github runner x)
------- Original Message -------
…On Saturday, August 6th, 2022 at 02:14, Andrew Innes ***@***.***> wrote:
Could this be caused by the test program retuning an error and then github hiding everything else?
Maybe try running it once where it's
run: |
test.exe E:/
exit 0
So that it doesn't error out and you still get the test output
—
Reply to this email directly, [view it on GitHub](#503 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AKFROC25ZJNIZMNQ45RTJKTVXWU6XANCNFSM542XZHWQ).
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
@maharmstone Hey, any news on this ? |
| mkdir upload | ||
| mkdir upload/${{ matrix.win_arch }} | ||
| copy src/btrfs.inf upload | ||
| copy build/ubtrfs.dll upload/${{ matrix.win_arch }} | ||
| copy build/shellbtrfs.dll upload/${{ matrix.win_arch }} | ||
| copy build/mkbtrfs.exe upload/${{ matrix.win_arch }} | ||
| copy build/btrfs.sys upload/${{ matrix.win_arch }} | ||
| mkdir upload-pdb | ||
| mkdir upload-pdb/${{ matrix.win_arch }} | ||
| copy build/ubtrfs.pdb upload-pdb/${{ matrix.win_arch }} | ||
| copy build/shellbtrfs.pdb upload-pdb/${{ matrix.win_arch }} | ||
| copy build/mkbtrfs.pdb upload-pdb/${{ matrix.win_arch }} | ||
| copy build/btrfs.pdb upload-pdb/${{ matrix.win_arch }} |
There was a problem hiding this comment.
| mkdir upload | |
| mkdir upload/${{ matrix.win_arch }} | |
| copy src/btrfs.inf upload | |
| copy build/ubtrfs.dll upload/${{ matrix.win_arch }} | |
| copy build/shellbtrfs.dll upload/${{ matrix.win_arch }} | |
| copy build/mkbtrfs.exe upload/${{ matrix.win_arch }} | |
| copy build/btrfs.sys upload/${{ matrix.win_arch }} | |
| mkdir upload-pdb | |
| mkdir upload-pdb/${{ matrix.win_arch }} | |
| copy build/ubtrfs.pdb upload-pdb/${{ matrix.win_arch }} | |
| copy build/shellbtrfs.pdb upload-pdb/${{ matrix.win_arch }} | |
| copy build/mkbtrfs.pdb upload-pdb/${{ matrix.win_arch }} | |
| copy build/btrfs.pdb upload-pdb/${{ matrix.win_arch }} | |
| '', '/${{ matrix.win_arch }}', '-pdb', '-pdb/${{ matrix.win_arch }}' | ForEach-Object { New-Item -ItemType Directory "upload$_" } | |
| '', '-vol' | ForEach-Object { Copy-Item "src/btrfs$_.inf" upload } | |
| 'ubtrfs.dll', 'shellbtrfs.dll', 'mkbtrfs.exe', 'btrfs.sys' | ForEach-Object { | |
| Copy-Item "build/$_" upload/${{ matrix.win_arch }} | |
| Copy-Item "build/$($_.Substring(0, $_.Length - 3))pdb" upload-pdb/${{ matrix.win_arch }} | |
| } |
Should be PowerShell (mkdir is not PowerShell), and should copy btrfs-vol.inf as well
There was a problem hiding this comment.
If your interest in this is because you are planning to run your own fork with different filename limits, do NOT do this. You will corrupt your filesystem when you use it on Linux, as even with the changes on that Russian site the kernel driver will overwrite other data with the long filename.
There was a problem hiding this comment.
Nah that's not my sole reason, I did try that. It didn't corrupt anything, but I did get a warning on Linux when running btrfs commands, renaming such paths removes any warnings, and just works.
My goal wasn't really to run a fork (branch is deleted now), but just to see if the CI worked, it didn't work well enough for me, so I learned how to do the signing manually.
Hey, following what I said in #499, I made a minimal CI setup
you can check https://github.com/iTrooz/btrfs/actions/runs/2748809550 for an example run
Unfortunately, I couldn't test the binaries because I didn't understand how to sign the driver
A few questions :
Do you want the CI to automatically sign the driver ? From what I understand, drivers have to be signed by a Microsoft-trusted CA, so I guess you have a key with such trust.
It could be added as a secret in the CI (See Settings -> Secrets -> Actions ) so the CI can use the key to sign the driver while keeping it secure.
This would allow people to test nightly versions of the driver (to test if their issue was resolved for example) without going through the hassle of signing it themselves
In case you do not want to add the key to the CI :
I understand that you did not agree for such a feature and that you may refuse it, I only worked on it because I had some free time and wanted to learn how to compile the driver