New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable cross-architecture installs using qemu #866

Closed
wants to merge 4 commits into
base: master
from

Conversation

Projects
None yet
5 participants
@npmccallum
Contributor

npmccallum commented Jul 12, 2017

These patches make it possible to use dnf within a series of commands to create a cross-architecture root. Basically, you install qemu-user-static into the root first, using the host architecture. Then you install the target packages you want using --setopt=arch=$ARCH. If you have set up this process properly, all scriptlets will be run using the emulator. This makes it possible to construct cross-architecture chroots and containers using dnf.

@Conan-Kudo

This looks great, but I'm not sure it's a good idea to ignore the architecture by default. We probably want this to be done only specifically for the container-root case.

Maybe it should be wired up as part of --installroot usage only? In normal cases, I would want the architecture to be validated.

@npmccallum

This comment has been minimized.

Show comment
Hide comment
@npmccallum

npmccallum Jul 13, 2017

Contributor

@Conan-Kudo I had the same thought. But it wasn't clear to me how to get this information into the place where we set the flag. Could someone more familiar with the code give me a hint?

Contributor

npmccallum commented Jul 13, 2017

@Conan-Kudo I had the same thought. But it wasn't clear to me how to get this information into the place where we set the flag. Could someone more familiar with the code give me a hint?

@npmccallum

This comment has been minimized.

Show comment
Hide comment
@npmccallum

npmccallum Jul 13, 2017

Contributor

@Conan-Kudo I figured it out. Here is a new patch set.

Contributor

npmccallum commented Jul 13, 2017

@Conan-Kudo I figured it out. Here is a new patch set.

@Conan-Kudo

Looks good to me.

@npmccallum

This comment has been minimized.

Show comment
Hide comment
@npmccallum

npmccallum Jul 13, 2017

Contributor

@Conan-Kudo I changed my mind over lunch. I think we want to enable RPMPROB_FILTER_IGNOREARCH in all cases. The reason for this is that inside the container/chroot we won't have any way to update the qemu-user-static package because we can't override the architecture back to the host architecture since we are emulated.

Contributor

npmccallum commented Jul 13, 2017

@Conan-Kudo I changed my mind over lunch. I think we want to enable RPMPROB_FILTER_IGNOREARCH in all cases. The reason for this is that inside the container/chroot we won't have any way to update the qemu-user-static package because we can't override the architecture back to the host architecture since we are emulated.

@Conan-Kudo

This comment has been minimized.

Show comment
Hide comment
@Conan-Kudo

Conan-Kudo Jul 13, 2017

Member

@npmccallum Then we should probably have an --ignorearch switch/flag, because this can't be the default behavior.

Member

Conan-Kudo commented Jul 13, 2017

@npmccallum Then we should probably have an --ignorearch switch/flag, because this can't be the default behavior.

@npmccallum

This comment has been minimized.

Show comment
Hide comment
@npmccallum

npmccallum Jul 13, 2017

Contributor

@Conan-Kudo This patch set implements something very close to what we talked about on IRC.

First, we enable setting basearch and arch as options. Second, we add a new option ignorearch which sets the RPM flag. Third, we add a new, synthetic, CLI argument (--forcearch=ARCH) which sets both the arch and ignorearch options.

All three options (basearch, arch, ignorearch) can thus be manipulated independently. But for the most common case on the CLI, only one argument will be used (--forcearch=ARCH). If users specify an invalid architecture, we also generate an appropriate error message documenting all the valid options:

$ dnf --forcearch=FOO
usage: dnf [-c [config file]] [-q] [-v] [--version] [--installroot [path]]
           [--nodocs] [--noplugins] [--enableplugin [plugin]]
           [--disableplugin [plugin]] [--releasever RELEASEVER]
           [--setopt SETOPTS] [--skip-broken] [-h] [--allowerasing] [-b] [-C]
           [-R [minutes]] [-d [debug level]] [--debugsolver]
           [--showduplicates] [-e ERRORLEVEL] [--obsoletes]
           [--rpmverbosity [debug level name]] [-y] [--assumeno]
           [--enablerepo [repo]] [--disablerepo [repo] | --repo [repo]]
           [-x [package]] [--disableexcludes [repo]]
           [--repofrompath [repo,path]] [--nogpgcheck] [--color COLOR]
           [--refresh] [-4] [-6] [--downloadonly] [--bugfix] [--enhancement]
           [--newpackage] [--security] [--advisory ADVISORY] [--bzs BUGZILLA]
           [--cves CVES] [--sec-severity {Critical,Important,Moderate,Low}]
           [--forcearch ARCH]
dnf: error: argument --forcearch: invalid choice: 'FOO' (choose from 'aarch64', 'alpha', 'alphaev4', 'alphaev45', 'alphaev5', 'alphaev56', 'alphaev6', 'alphaev67', 'alphaev68', 'alphaev7', 'alphapca56', 'amd64', 'armv5tejl', 'armv5tel', 'armv5tl', 'armv6hl', 'armv6l', 'armv7hl', 'armv7hnl', 'armv7l', 'armv8l', 'athlon', 'geode', 'i386', 'i486', 'i586', 'i686', 'ia32e', 'ia64', 'mips', 'mips64', 'mips64el', 'mipsel', 'noarch', 'ppc', 'ppc64', 'ppc64iseries', 'ppc64le', 'ppc64p7', 'ppc64pseries', 'riscv128', 'riscv32', 'riscv64', 's390', 's390x', 'sh3', 'sh4', 'sh4a', 'sparc', 'sparc64', 'sparc64v', 'sparcv8', 'sparcv9', 'sparcv9v', 'x86_64')
Contributor

npmccallum commented Jul 13, 2017

@Conan-Kudo This patch set implements something very close to what we talked about on IRC.

First, we enable setting basearch and arch as options. Second, we add a new option ignorearch which sets the RPM flag. Third, we add a new, synthetic, CLI argument (--forcearch=ARCH) which sets both the arch and ignorearch options.

All three options (basearch, arch, ignorearch) can thus be manipulated independently. But for the most common case on the CLI, only one argument will be used (--forcearch=ARCH). If users specify an invalid architecture, we also generate an appropriate error message documenting all the valid options:

$ dnf --forcearch=FOO
usage: dnf [-c [config file]] [-q] [-v] [--version] [--installroot [path]]
           [--nodocs] [--noplugins] [--enableplugin [plugin]]
           [--disableplugin [plugin]] [--releasever RELEASEVER]
           [--setopt SETOPTS] [--skip-broken] [-h] [--allowerasing] [-b] [-C]
           [-R [minutes]] [-d [debug level]] [--debugsolver]
           [--showduplicates] [-e ERRORLEVEL] [--obsoletes]
           [--rpmverbosity [debug level name]] [-y] [--assumeno]
           [--enablerepo [repo]] [--disablerepo [repo] | --repo [repo]]
           [-x [package]] [--disableexcludes [repo]]
           [--repofrompath [repo,path]] [--nogpgcheck] [--color COLOR]
           [--refresh] [-4] [-6] [--downloadonly] [--bugfix] [--enhancement]
           [--newpackage] [--security] [--advisory ADVISORY] [--bzs BUGZILLA]
           [--cves CVES] [--sec-severity {Critical,Important,Moderate,Low}]
           [--forcearch ARCH]
dnf: error: argument --forcearch: invalid choice: 'FOO' (choose from 'aarch64', 'alpha', 'alphaev4', 'alphaev45', 'alphaev5', 'alphaev56', 'alphaev6', 'alphaev67', 'alphaev68', 'alphaev7', 'alphapca56', 'amd64', 'armv5tejl', 'armv5tel', 'armv5tl', 'armv6hl', 'armv6l', 'armv7hl', 'armv7hnl', 'armv7l', 'armv8l', 'athlon', 'geode', 'i386', 'i486', 'i586', 'i686', 'ia32e', 'ia64', 'mips', 'mips64', 'mips64el', 'mipsel', 'noarch', 'ppc', 'ppc64', 'ppc64iseries', 'ppc64le', 'ppc64p7', 'ppc64pseries', 'riscv128', 'riscv32', 'riscv64', 's390', 's390x', 'sh3', 'sh4', 'sh4a', 'sparc', 'sparc64', 'sparc64v', 'sparcv8', 'sparcv9', 'sparcv9v', 'x86_64')
@Conan-Kudo

Looks good to me!

@Conan-Kudo

This comment has been minimized.

Show comment
Hide comment
@Conan-Kudo

Conan-Kudo Jul 13, 2017

Member

@npmccallum Can you add the information about these new things to the documentation? It's in the doc/ sub-directory.

Member

Conan-Kudo commented Jul 13, 2017

@npmccallum Can you add the information about these new things to the documentation? It's in the doc/ sub-directory.

npmccallum added some commits Jul 12, 2017

Provide an API for setting 'arch' and 'basearch'
This patch is mostly a rework of the original patch by Hristo Venev
<hristo@venev.name>.

RhBug:1190854
Add 'ignorearch' option
The 'ignorearch' option defaults to False (which implements the
current behavior). When it is set to True, we enable the
RPMPROB_FILTER_IGNOREARCH flag for RPM transactions. This enables DNF
to install packages for architectures that are different than the host
architecture.
Add the --forcearch CLI flag
This flag is synthetic (it doesn't set a configuration value
directly). Instead, when this flag is activated, it sets the 'arch'
and 'ignorearch' configuration options. This argument takes a
string value that is one of the value architectures. We enforce that a
valid architecture is specified by using the 'choices' parameter.

RhBug:1427604
@npmccallum

This comment has been minimized.

Show comment
Hide comment
@npmccallum

npmccallum Jul 14, 2017

Contributor

@ignatenkobrain All tests pass and @Conan-Kudo has approved the changes. Can we merge this?

Also, should we make a Fedora change proposal to advertise this functionality?

Contributor

npmccallum commented Jul 14, 2017

@ignatenkobrain All tests pass and @Conan-Kudo has approved the changes. Can we merge this?

Also, should we make a Fedora change proposal to advertise this functionality?

@npmccallum

This comment has been minimized.

Show comment
Hide comment
@npmccallum

npmccallum Jul 14, 2017

Contributor

@dmach Do you know when you'll be able to review this?

Contributor

npmccallum commented Jul 14, 2017

@dmach Do you know when you'll be able to review this?

@Conan-Kudo

This comment has been minimized.

Show comment
Hide comment
@Conan-Kudo

Conan-Kudo Jul 16, 2017

Member

@dmach Could you please review this ASAP? It's a pretty easy PR to review, and the functionality is quite desirable for Fedora and Mageia users.

Member

Conan-Kudo commented Jul 16, 2017

@dmach Could you please review this ASAP? It's a pretty easy PR to review, and the functionality is quite desirable for Fedora and Mageia users.

@dmach

This comment has been minimized.

Show comment
Hide comment
@dmach

dmach Jul 19, 2017

Contributor

The code looks good, no objections to the coding style or the code itself.
I'd only appreciate one change - add a detailed example on how to use --forcearch with qemu to make the use more straightforward for the users (me including).
Although I'm not going to block the pull request on such detail, we can always update the docs later.

Contributor

dmach commented Jul 19, 2017

The code looks good, no objections to the coding style or the code itself.
I'd only appreciate one change - add a detailed example on how to use --forcearch with qemu to make the use more straightforward for the users (me including).
Although I'm not going to block the pull request on such detail, we can always update the docs later.

@dmach

This comment has been minimized.

Show comment
Hide comment
@dmach

dmach Jul 19, 2017

Contributor
Contributor

dmach commented Jul 19, 2017

@dnf-bot

This comment has been minimized.

Show comment
Hide comment
@dnf-bot

dnf-bot Jul 19, 2017

Member

📌 Commit 1dfc57c has been approved by dmach

Member

dnf-bot commented Jul 19, 2017

📌 Commit 1dfc57c has been approved by dmach

@dnf-bot

This comment has been minimized.

Show comment
Hide comment
@dnf-bot

dnf-bot Jul 19, 2017

Member

⌛️ Testing commit 1dfc57c with merge b821687...

Member

dnf-bot commented Jul 19, 2017

⌛️ Testing commit 1dfc57c with merge b821687...

dnf-bot pushed a commit that referenced this pull request Jul 19, 2017

Add 'ignorearch' option
The 'ignorearch' option defaults to False (which implements the
current behavior). When it is set to True, we enable the
RPMPROB_FILTER_IGNOREARCH flag for RPM transactions. This enables DNF
to install packages for architectures that are different than the host
architecture.

Closes: #866
Approved by: dmach

dnf-bot pushed a commit that referenced this pull request Jul 19, 2017

Add the --forcearch CLI flag
This flag is synthetic (it doesn't set a configuration value
directly). Instead, when this flag is activated, it sets the 'arch'
and 'ignorearch' configuration options. This argument takes a
string value that is one of the value architectures. We enforce that a
valid architecture is specified by using the 'choices' parameter.

RhBug:1427604

Closes: #866
Approved by: dmach

dnf-bot pushed a commit that referenced this pull request Jul 19, 2017

Add myself to the AUTHORS file
Closes: #866
Approved by: dmach
@dnf-bot

This comment has been minimized.

Show comment
Hide comment
@dnf-bot

dnf-bot Jul 19, 2017

Member

☀️ Test successful - status-jenkins
Approved by: dmach
Pushing b821687 to master...

Member

dnf-bot commented Jul 19, 2017

☀️ Test successful - status-jenkins
Approved by: dmach
Pushing b821687 to master...

@dnf-bot dnf-bot closed this in 9498404 Jul 19, 2017

@npmccallum

This comment has been minimized.

Show comment
Hide comment
@npmccallum

npmccallum Jul 19, 2017

Contributor

@dmach You can test this today on Fedora 26 (plus this patchset).

$ sudo dnf -y install qemu-user-static
...
$ sudo dnf -y --installroot=/tmp/foo --repo=fedora --repo=updates --forcearch=aarch64 install @core
...
$ sudo chroot /tmp/foo /usr/bin/uname -r -m
4.10.0-1.fc26.x86_64 aarch64
Contributor

npmccallum commented Jul 19, 2017

@dmach You can test this today on Fedora 26 (plus this patchset).

$ sudo dnf -y install qemu-user-static
...
$ sudo dnf -y --installroot=/tmp/foo --repo=fedora --repo=updates --forcearch=aarch64 install @core
...
$ sudo chroot /tmp/foo /usr/bin/uname -r -m
4.10.0-1.fc26.x86_64 aarch64
@dmach

This comment has been minimized.

Show comment
Hide comment
@dmach

dmach Jul 20, 2017

Contributor

@npmccallum the steps you provided don't work and I discovered why:
You need to reboot after installing qemu-user-static.
Maybe there's a way how to avoid reboot, but I did not go that far...

Reason:

$ man systemd-binfmt
...
DESCRIPTION
       systemd-binfmt.service is an early boot service that registers additional binary formats for executables in the kernel.
Contributor

dmach commented Jul 20, 2017

@npmccallum the steps you provided don't work and I discovered why:
You need to reboot after installing qemu-user-static.
Maybe there's a way how to avoid reboot, but I did not go that far...

Reason:

$ man systemd-binfmt
...
DESCRIPTION
       systemd-binfmt.service is an early boot service that registers additional binary formats for executables in the kernel.
@npmccallum

This comment has been minimized.

Show comment
Hide comment
@npmccallum

npmccallum Jul 20, 2017

Contributor

No need to reboot. Just do: $ sudo systemctl restart systemd-binfmt.service

This is one of the bugs in the qemu package I'm trying to get fixed.

Contributor

npmccallum commented Jul 20, 2017

No need to reboot. Just do: $ sudo systemctl restart systemd-binfmt.service

This is one of the bugs in the qemu package I'm trying to get fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment