Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run binary package generation and file classification in parallel threads #695

Merged
merged 9 commits into from
May 21, 2019
Merged

Run binary package generation and file classification in parallel threads #695

merged 9 commits into from
May 21, 2019

Commits on May 14, 2019

  1. Reindent packageBinary() to common rpm style, trim trailing whitespace 

    Cosmetics only.
    @pmatilai
    pmatilai committed May 14, 2019
    Configuration menu
    Copy the full SHA
    0bbac87 View commit details
    Browse the repository at this point in the history

  2. Refactor filename generation and dir check out of packageBinary() 

    These are clearly separate operations that deserve functions of their
own, and this makes memory management nicer. In addition, in case
the directory creation fails we now actually error out instead
of trying to continue, and take care not to fail in case somebody
created it behind our back. No other functional changes though.
    @pmatilai
    pmatilai committed May 14, 2019
    Configuration menu
    Copy the full SHA
    261fe54 View commit details
    Browse the repository at this point in the history

  3. Remember the filename of built packages in the package struct 

    No functional changes and doesn't make much difference here, but we'll
need this later.
    @pmatilai
    pmatilai committed May 14, 2019
    Configuration menu
    Copy the full SHA
    179690f View commit details
    Browse the repository at this point in the history

  4. Refactor package set checking out of packageBinaries() 

    Now that we can, split the "lets run something on generated packages"
check out of packageBinaries(), it doesn't belong there at all. No
functional changes other fix attempt to check non-built packages
which return with no filename but RPMRC_OK from packageBinary().
    @pmatilai
    pmatilai committed May 14, 2019
    Configuration menu
    Copy the full SHA
    22e1805 View commit details
    Browse the repository at this point in the history

  5. Remember package creation result in packages themselves 

    No functional change, but we'll need this later.
    @pmatilai
    pmatilai committed May 14, 2019
    Configuration menu
    Copy the full SHA
    ec1a0e0 View commit details
    Browse the repository at this point in the history

  6. Copy changelogs from src.rpm, not first binary package 

    The comment has been wrong for more than twenty years...
No functional changes here, but this will make a difference in the
next commits.
    @pmatilai
    pmatilai committed May 14, 2019
    Configuration menu
    Copy the full SHA
    e3b2ca0 View commit details
    Browse the repository at this point in the history

  7. Add infrastructure for OpenMP support in librpmbuild 

    Enable OpenMP use in librpmbuild and set the number of OMP threads
from rpm config after spec parsing. The place matters as we want to
allow individual specs to control and disable parallel builds.
    @pmatilai
    pmatilai committed May 14, 2019
    Configuration menu
    Copy the full SHA
    a9462af View commit details
    Browse the repository at this point in the history

  8. Run binary package generation in parallel threads 

    It's worth noting that this really is walking on thin ice as only few
parts of rpm are thread-protected. The spec is entirely unprotected so
must be accessed only for read-only purposes from parallel jobs (and
we should work towards enforcing that via const-correctness and other
protection as needed), and similarly the package struct and headers
are unprotected so they can only be manipulated within a single thread.

Based on initial work by Alexander Kanavin in PR rpm-software-management#226
    @pmatilai
    pmatilai committed May 14, 2019
    Configuration menu
    Copy the full SHA
    fabfb6b View commit details
    Browse the repository at this point in the history

  9. Run file classification in parallel threads 

    Our file classification is not exactly in large quantities. Fedora's
kernel-debuginfo-common has circa 25000 files, and classifying them
serially takes about a minute on my rusty old T520. With parallelization
this goes down to ~24s.

It's all remarkably simple, except for the fact that libmagic is not
thread-safe so we need separate magic handles for each of our threads.
This will leak those libmagic handles on error situations, I don't see
any obvious, nice way to handle that.
    @pmatilai
    pmatilai committed May 14, 2019
    Configuration menu
    Copy the full SHA
    94e5708 View commit details
    Browse the repository at this point in the history