New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for Fedora or RHEL Images pt 2 #276

Merged
merged 12 commits into from Jan 16, 2019

Conversation

Projects
None yet
5 participants
@dustymabe
Copy link
Collaborator

dustymabe commented Jan 11, 2019

this builds on #275

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

Still haven't fully tested this out yet but I'd like to get up this PR and we can work through any issues in the morning.

@dustymabe dustymabe added the WIP label Jan 11, 2019

@miabbott

This comment has been minimized.

Copy link
Contributor

miabbott commented Jan 11, 2019

I built this locally, but ran into an error trying to use the container itself:

$ git rev-parse HEAD
70e151d9106a931656e680784b45fa061f2ceb21
$ sudo buildah bud -t miabbott/coreos-assembler:rhel -f Dockerfile.rhel .

...lots of packages installed...

STEP 15: ENTRYPOINT ["/usr/bin/dumb-init", "scl", "enable", "rh-python36", "/usr/bin/coreos-assembler"]                                                                                                                                                                                  
ERRO[1704] HOSTNAME is not supported for OCI image format, hostname 1d561c58fd2b will be ignored. Must use `docker` format      
STEP 16: COMMIT containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,overlay.override_kernel_check=true]localhost/miabbott/coreos-assembler:rhel                                                                                   
Getting image source signatures                                                                                                                                                                                                                                                          
Skipping fetch of repeat blob sha256:56a763045c4544c21c458f3cd948a46384e4b12f9deacd1aede445af598f6d84                                                                                                                                                                                    
Skipping fetch of repeat blob sha256:ab9227d97750aff30bf47631468e9b69dddcdd4af6a853233d6288d05770fcf8                                                                                                                                                                                    
Copying blob sha256:1228fe65d83cb4858d659bb79c228fbdd5737385ceb8cba11547699ef3a604b0                                                                                                                                                                                                     
 541.52 MiB / 541.52 MiB [=================================================] 25s                                                   
Copying config sha256:cd923464bb1ce7b294f4b2e54d26560100e5bb18fe1b6b92cba1320f9b9a8b81                                          
 3.17 KiB / 3.17 KiB [======================================================] 0s                                                
Writing manifest to image destination                                                                                                                      
Storing signatures                                                                                                             
--> cd923464bb1ce7b294f4b2e54d26560100e5bb18fe1b6b92cba1320f9b9a8b81                                        

$ sudo podman images | grep miabbott/coreos-assembler
localhost/miabbott/coreos-assembler                                             rhel     cd923464bb1c   30 minutes ago   1.92 GB

$ alias cosa-rhel='sudo podman run --rm --net=host -ti --privileged --userns=host -v $PWD:/srv localhost/miabbott/coreos-assembler:rhel'

$ cosa-rhel init --force https://gitlab.cee.redhat.com/coreos/redhat-coreos.git maipo
[sudo] password for miabbott:
Unable to open /etc/scl/conf//usr/bin/coreos-assembler!
@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

thanks @miabbott - i'm just now getting to actually testing this myself. had some selinux issues on the podman build that caused me some pain.. finally got everything labeled up right and building again now to run through a build.

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

ok think I figured it out.. scl was interpreting everyting up to the last arg as other SCLs to enable. we can explicitly tell it to stop considering args as SCLs with -- like so:

ENTRYPOINT ["/usr/bin/dumb-init", "scl", "enable", "rh-python36", "--", "/usr/bin/coreos-assembler"]
@miabbott

This comment has been minimized.

Copy link
Contributor

miabbott commented Jan 11, 2019

@ashcrow suggested closing #275 and focusing dev efforts here.

To that end, we're gonna end up with some missing dependencies:

$ cosa-rhel init --force https://gitlab.cee.redhat.com/coreos/redhat-coreos.git maipo
fatal: Failed to find expected dependencies:  dnf-utils libguestfs-tools python2-gobject-base python3-gobject-base
@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

yeah I see this when building:

No package dnf-utils available.           
No package python2-gobject-base available.

we can pick/choose deps based on platform now, so that's fine. we just need to make sure we gather everything we actually need to do a build

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

looks like libguestfs-tools is getting removed when grubby gets removed

================================================================================
 Package               Arch      Version                 Repository        Size
================================================================================
Removing:                                                                   
 grubby                x86_64    8.28-25.el7             @maipo-server    124 k 
Removing for dependencies:                                          
 kernel                x86_64    3.10.0-957.1.3.el7      @maipo-server     63 M
 libguestfs            x86_64    1:1.38.2-12.el7_6.1     @maipo-server    4.3 M
 libguestfs-tools      noarch    1:1.38.2-12.el7_6.1     @maipo-server     31 k
 libguestfs-tools-c    x86_64    1:1.38.2-12.el7_6.1     @maipo-server     19 M
 perl-Sys-Guestfs      x86_64    1:1.38.2-12.el7_6.1     @maipo-server    1.1 M
                                                             
Transaction Summary                                             
================================================================================
Remove  1 Package (+5 Dependent packages)
@ashcrow

This comment has been minimized.

Copy link
Collaborator

ashcrow commented Jan 11, 2019

looks like libguestfs-tools is getting removed when grubby gets removed

Good catch!

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

ok got this mostly worked out I think.. the one tricky one is python2-gobject-base - do we even need that since we are working on python3?

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

moving python2-gobject-base to be a Fedora only dep for now and seeing if everything works

@miabbott

This comment has been minimized.

Copy link
Contributor

miabbott commented Jan 11, 2019

In Fedora:

$ rpm -q --whatrequires python2-gobject-base
virt-manager-common-1.5.1-1.fc28.noarch
@miabbott

This comment has been minimized.

Copy link
Contributor

miabbott commented Jan 11, 2019

In RHEL:

$ rpm -q --requires virt-manager-common
genisoimage
libosinfo >= 0.2.11
libvirt-python >= 0.7.0
libxml2-python
pygobject3-base
python-ipaddr
python-requests
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PartialHardlinkSets) <= 4.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1
@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

now I'm getting this error during init:

+ virt-format --filesystem=xfs -a cache/cache.qcow2 
libguestfs: error: mkfs: xfs: /dev/sda1: mkfs.xfs: No such file or directory

I thought it was because xfsprogs was not installed in the container, so I added it. still seeing the same issue

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

ok switched to ext4.. now seeing:

$ which cass
alias cass='sudo podman run --rm -ti -v ${PWD}:/srv/ ${COREOS_ASSEMBLER_CONFIG_GIT:+-v  $COREOS_ASSEMBLER_CONFIG_GIT:/srv/src/config/:ro} ${COREOS_ASSEMBLER_GIT:+-v $COREOS_ASSEMBLER_GIT/src/:/usr/lib/coreos-assembler/:ro} --workdir /srv --device /dev/kvm cael7e'
$ cass fetch
info: Missing CAP_SYS_ADMIN; using virt
Using manifest: /srv/src/config/manifest.yaml
bwrap: No permissions to creating new namespace, likely because the kernel does not allow non-privileged user namespaces. On e.g. debian this can be enabled with 'sysctl kernel.unprivileged_userns_clone=1'.
error: bwrap test failed, see <https://github.com/projectatomic/rpm-ostree/pull/429>: Executing bwrap(true): Child process killed by signal 1
$ cass shell
[coreos-assembler]$ rpm -q rpm-ostree ostree
rpm-ostree-2018.5-2.atomic.el7.x86_64
ostree-2018.5-1.el7.x86_64
@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

ok pushed up another commit.. this is what I've got and i'm stuck on #276 (comment)

@jlebon

This comment has been minimized.

Copy link
Member

jlebon commented Jan 11, 2019

Ahhh, I think that's an old bug where it required privs even just to print the manifest.
Anyway right now, coreos-assembler pretty much requires v2018.10. I can get those builds going.

@dustymabe dustymabe force-pushed the dustymabe:dusty-scl branch from 118cefc to 6831c4a Jan 11, 2019

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 11, 2019

ok pushed up another commit..

and.. just amended that commit since I updated the submodule too.. the commit now does not touch the mantle submodule. forced pushed the amended commit

@@ -90,7 +90,10 @@ install_rpms() {

# Commented out for now, see above
#dnf remove -y $builddeps}
rpm -q grubby && yum remove -y grubby
# can't remove grubby on el7 because libguestfs-tools depends on it

This comment has been minimized.

@miabbott

miabbott Jan 11, 2019

Contributor

Dang, I was going to suggest we should at least remove kernel on RHEL, but libguestfs-tools-c requires it. 🙄

@@ -180,5 +180,6 @@ mkdir -p tmp
ostree --repo=repo init --mode=archive
if ! has_privileges && [ ! -f cache/cache.qcow2 ]; then
qemu-img create -f qcow2 cache/cache.qcow2 10G
LIBGUESTFS_BACKEND=direct virt-format --filesystem=xfs -a cache/cache.qcow2
# use ext4 because we hit an error on el7 when trying to use xfs

This comment has been minimized.

@miabbott

miabbott Jan 11, 2019

Contributor

https://bugzilla.redhat.com/show_bug.cgi?id=1123221

I confirmed that this can be solved with an install of libguestfs-xfs

This comment has been minimized.

@dustymabe

dustymabe Jan 11, 2019

Collaborator

ahh. cool. will fix that up after lunch

@dustymabe dustymabe force-pushed the dustymabe:dusty-scl branch from 6831c4a to 9a5688f Jan 11, 2019

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 14, 2019

ok with new ostree/rpm-ostree i'm getting farther, but still hitting some papercuts..

  • -C is not an option to git in EL7
  • --prepare is not an option to supermin in EL7

i'm sure there will be more

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 14, 2019

* `--prepare` is not an option to supermin in EL7

ok ignoring supermin (since we are running in privileged mode on EL7) I get an ostree compose, but my virt-install invocation is not working.. still looking at it.

@cgwalters

This comment has been minimized.

Copy link
Member

cgwalters commented Jan 14, 2019

Would it help to target EL8 instead of 7?

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 14, 2019

Would it help to target EL8 instead of 7?

probably.. if we hit more blockers and ootpa is an option then we can explore that.. I feel like we are getting close so let's see where this goes in the next little bit and then evaluate.

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 14, 2019

ok the next thing I'm hitting is the fact that we are using 9pfs for the anaconda install, which doesn't work with RHEL at all.

@cgwalters

This comment has been minimized.

Copy link
Member

cgwalters commented Jan 14, 2019

ok the next thing I'm hitting is the fact that we are using 9pfs for the anaconda install, which doesn't work with RHEL at all.

Yeah 😢 - we may need to fall back to NFS or something awful. That said there is actually a solution on the horizon.

@cgwalters

This comment has been minimized.

Copy link
Member

cgwalters commented Jan 14, 2019

Yeah cry - we may need to fall back to NFS or something awful.

Or I guess, rely on this being fixed indirectly via the "don't use anaconda" code path?

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 16, 2019

One thing I noticed is that the python3 -m http.server that is started spews all logs to stdout; so every file that is fetched gets a line printed out.

yep. i don't know what is worse? having all the logs, or not having any logs

@ashcrow

This comment has been minimized.

Copy link
Collaborator

ashcrow commented Jan 16, 2019

One thing I noticed is that the python3 -m http.server that is started spews all logs to stdout; so every file that is fetched gets a line printed out.

yep. i don't know what is worse? having all the logs, or not having any logs

Not having any IMHO

@ashcrow
Copy link
Collaborator

ashcrow left a comment

Overall 👍 ... 1 request and 1 question:

Request: Commit Dockerfile.rhel: fix scl invovation ➡️ Dockerfile.rhel: fix scl invocation
Question: I don't see it as an issue, but I'm curious why the switch to serving the content over http?

@miabbott

This comment has been minimized.

Copy link
Contributor

miabbott commented Jan 16, 2019

I agree with @ashcrow that this overall LGTM.

Though, I'm a bit concerned with the lack of run support, which means we also can't run the kola qemu tests in the container. For the use case of the RHCOS pipeline, this could be another problem we have to solve. Maybe it means building a kola container from RHEL bits that support qemu usage.

@ashcrow

This comment has been minimized.

Copy link
Collaborator

ashcrow commented Jan 16, 2019

If we can run the tests from the Fedora container I think that would be fine ... but building would need to be from RHEL.

@dustymabe dustymabe force-pushed the dustymabe:dusty-scl branch from 229a429 to 0fc2e17 Jan 16, 2019

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 16, 2019

Request: Commit Dockerfile.rhel: fix scl invovation arrow_right Dockerfile.rhel: fix scl invocation

fixed

Question: I don't see it as an issue, but I'm curious why the switch to serving the content over http?

we were previously using 9pfs for this but the RHEL kernel and the RHEL packages don't support it (it has been unreliable in the past and a nightmare to support so they probably ripped it out for good reason). So we had to switch to something else. See commit message in 0fc2e17

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 16, 2019

Though, I'm a bit concerned with the lack of run support

I think we can probably enable run but maybe let's do it in a follow up PR. I forget what the issues were right now but I think one of them is related to qemu-kvm not being in the $PATH

ashcrow and others added some commits Jan 7, 2019

Makefile: Explicit directory creation
On some distributions with older versions of the install command
explicit creation of full paths seem required.

Signed-off-by: Steve Milner <smilner@redhat.com>
rhel: RHEL version of c-a building
Signed-off-by: Steve Milner <smilner@redhat.com>
cleanup for supporting EL7
- deduped some of the code in the previous commits
- hopefully made it so we don't need a separate scl
  wrapper script
Dockerfile.rhel: fix scl invocation
`scl` was interpreting everything up to the last arg as other SCLs to
enable. We can explicitly tell it to stop considering args as SCLs with
`--`.
a few more fixups for el7
- fixup a few dependencies
- user filter in runtime dep checks
- add libguestfs-xfs for virt-make-fs with xfs on el7
- don't remove grubby (removes libguestfs-tools)
export ISFEDORA/ISEL for extended use
In child processes, etc.
disallow unpriv mode on EL7
The supermin args have changed enough such that it's not worth trying
to make unpriv mode work on EL7. Since we're not using unpriv mode there
today just hard require priv for now.
convert python files to use `#!/usr/bin/env python3`
This will cause them to use the SCL on el7. It does make it
so that `python3 -u` on the shebang line doesn't work. We work
around this by setting PYTHONUNBUFFERED env var.

There is one file where we stick with using system python because
the SCL in EL7 doesn't have gobject library.
gf-oemid: disable qemu_wrapper on EL7
Not working there for some reason
prune: fix timestamp comparison bug, add log
We were inapropriately pulling `build-timestamp` from the metadata
and not `coreos-assembler.build-timestamp`. We probably didn't see
this for so long because you often have a new ostree when a new build
happens so the `ostree-timestamp` fallback was good enough.

Also add a log statement that is useful/informative.
virt-install: workaround limitations on EL7
- 9pfs doesn't work on EL7 so I converted our code to use an
  http server for sharing the ostree into anaconda
- virt-install --console=log.file is not supported on EL7 so
  use the serial console on EL7 if we can

@dustymabe dustymabe force-pushed the dustymabe:dusty-scl branch from 0fc2e17 to dec465c Jan 16, 2019

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 16, 2019

now rebased on top of latest master

@miabbott

This comment has been minimized.

Copy link
Contributor

miabbott commented Jan 16, 2019

Were we going to drop the ShellCheck requirement for EL7?

@dustymabe

This comment has been minimized.

Copy link
Collaborator

dustymabe commented Jan 16, 2019

Were we going to drop the ShellCheck requirement for EL7?

seems reasonable to me.. preferably in a separate PR

@ashcrow
Copy link
Collaborator

ashcrow left a comment

LGTM

@dustymabe dustymabe removed the WIP label Jan 16, 2019

@dustymabe dustymabe changed the title WIP: Allow for Fedora or RHEL Images pt 2 Allow for Fedora or RHEL Images pt 2 Jan 16, 2019

@miabbott
Copy link
Contributor

miabbott left a comment

There's some follow-on work to be done, but it's possible to build a RHEL-based cosa, so let's get it merged!

@ashcrow ashcrow merged commit a1fa1ed into coreos:master Jan 16, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment