Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run binary package generation via thread pools #226

Closed
wants to merge 4 commits into from

Conversation

@kanavin
Copy link
Contributor

kanavin commented Jun 1, 2017

Please see here for background information and rationale:
#211

Caveats:

  • this work was based on an older commit from master that Yocto is currently using. I have rebased it to latest upstream master, but did not test that the rebased version still works. People have asked me to publish the code as quickly as possible, so here it is. I guess this is more for evaluating the general approach and suitability for merging into upstream.
@proyvind

This comment has been minimized.

Copy link
Contributor

proyvind commented Jun 1, 2017

cool!

I'll try get around reviewing it myself at least when I find the time, then provide feedback and/or my approval of (FWIW). :)

@@ -14,6 +14,8 @@
#include <rpm/rpmfileutil.h>
#include <rpm/rpmlog.h>

#include <prtpool.h> /* NSPR thread pools */

This comment has been minimized.

Copy link
@ignatenkobrain

ignatenkobrain Jun 1, 2017

Member

I'm kinda reluctant for such external dependencies

This comment has been minimized.

Copy link
@kanavin

kanavin Jun 1, 2017

Author Contributor

rpm already has a hard dependency on nss, and nss has a hard dependency on nspr, so it seemed obvious to utilize the thread pool API from nspr. Why you are reluctant?

This comment has been minimized.

Copy link
@ignatenkobrain

ignatenkobrain Jun 1, 2017

Member

@kanavin NSS is not hard dependency, it has support for openssl and other crypto backend and I would say that NSS is hardest to bootstrap so we swithced to OpenSSL in Fedora not so long time ago... So assuming NSS to be present is very bad idea.

This comment has been minimized.

Copy link
@Conan-Kudo

Conan-Kudo Jun 1, 2017

Member

It'd be pretty hard for me to compile this on Mac OS now, as the Netscape stuff is difficult to bootstrap.

This comment has been minimized.

Copy link
@kanavin

kanavin Jun 1, 2017

Author Contributor

Right, openssl support is very recent, so I missed that. Do you have suggestions on what could provide a good thread pool API, if not nspr? I wouldn't want to do a poor internal re-implementation on top of pthreads.

The whole thing can also be made conditional.

This comment has been minimized.

Copy link
@ignatenkobrain

ignatenkobrain Jun 1, 2017

Member

openmp?

This comment has been minimized.

Copy link
@proyvind

proyvind Jun 1, 2017

Contributor

openmp, unless for some reason would turn out to be unfeasiable, would be the definitive candidate IMO given it's standardization and implementation available from all compilers of relevance. 👍

This comment has been minimized.

Copy link
@kanavin

kanavin Jun 5, 2017

Author Contributor

Alright, I'll rework the patches with openmp, and re-do the pull request.

@kanavin kanavin force-pushed the kanavin:parallel-rpm branch from f1e7a83 to 1579d7e Jun 8, 2017
@kanavin

This comment has been minimized.

Copy link
Contributor Author

kanavin commented Jun 8, 2017

There you go, I force pushed the reworked patches. It is 100% pure openmp now :)

@kanavin kanavin force-pushed the kanavin:parallel-rpm branch from 1579d7e to c0fdce3 Jun 9, 2017
@proyvind

This comment has been minimized.

Copy link
Contributor

proyvind commented Jun 9, 2017

You even caught some design shortcomings of rpm. and improved on poor implementations, bravo!

While only having quickly reviewed code, (but having knowledge of openmp and code in question at leastt), only thing I can come to think of is more aestetics, ie.

  • #pragma omp critical(rpmstrpool)
  • {
    if (pool && sid > 0 && sid <= pool->offs_size) {
    slen = strlen(pool->offs[sid]);
    }
  • }
#pragma omp critical(rpmstrpool) {
    if (pool && sid > 0 && sid <= pool->offs_size) {
    slen = strlen(pool->offs[sid]);
    }
}

would be more readable IMO.

But wars can't possible have started by less, so I give my approval after giving a first sight. 👍 :)

@kanavin

This comment has been minimized.

Copy link
Contributor Author

kanavin commented Jun 9, 2017

The reason I didn't change the indentation when wrapping the code in those pragmas was to keep the patch more readable.

Also, don't merge these just yet: they work flawlessly on my machine, but I get reports that they crash on our automated build system. It may be an issue with an older gcc version. Obviously I'd like you to try them and report :)

EDIT: I also broke the last patch when rebasing it to master, sorry about that. So look at the code, but don't try to build :)

@kanavin kanavin force-pushed the kanavin:parallel-rpm branch from c0fdce3 to 12d59d3 Jun 13, 2017
@kanavin

This comment has been minimized.

Copy link
Contributor Author

kanavin commented Jun 13, 2017

I have now updated the patchset again, addressing the crashes with old gcc, and Jeff's comment above. And this time they should be rebased correctly :)

@pmatilai

This comment has been minimized.

Copy link
Contributor

pmatilai commented Jun 13, 2017

I've no prior knowledge of OMP but doesn't look half bad on first sight.

Cosmetics aside, we can't really have half the codebase doing half-assed manual pthread locking here and there and another half using OMP, AIUI this is undefined behavior and the string pool is heavily used by non-build code as well.

So to set expectations straight: we'll look at merging after rpm 4.14 is branched off, which will be at least couple of months away still.

@Conan-Kudo

This comment has been minimized.

Copy link
Member

Conan-Kudo commented Jun 13, 2017

This seems to work correctly on my Mac OS X 10.10 system, though admittedly I don't have a great test case with lots of subpackages. With ~3 subpackages (progs, libs, devel), it seems to work fine.

@pmatilai

This comment has been minimized.

Copy link
Contributor

pmatilai commented Jun 13, 2017

BTW it'd probably be easier and less intrusive to make buildtime and buildhost part of the spec struct, initialized in newSpec() or so. AFAICS all the relevant places are receiving spec as the argument already, with the exception of writeRPM().

@ignatenkobrain

This comment has been minimized.

Copy link
Member

ignatenkobrain commented Jul 30, 2017

@kanavin any news here?

@kanavin

This comment has been minimized.

Copy link
Contributor Author

kanavin commented Jul 31, 2017

@ignatenkobrain The patches have been merged to Yocto project:
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3

As @pmatilai said he will not look at them until after 4.14 is ready (which is two months away), right now I am not in a particular hurry to rebase and update them. But I haven't forgotten :) Is there anything specific you would like me to do now?

@Conan-Kudo

This comment has been minimized.

Copy link
Member

Conan-Kudo commented Aug 16, 2017

@kanavin Now that rpm 4.14 has been branched, could you rebase them for review to merge into master?

kanavin added 3 commits May 25, 2017
So that it can be run as a thread pool task.
Otherwise multithreaded rpm building explodes in various ways due
to data races.
@kanavin kanavin force-pushed the kanavin:parallel-rpm branch from 12d59d3 to 8d9c63d Aug 29, 2017
@kanavin

This comment has been minimized.

Copy link
Contributor Author

kanavin commented Aug 29, 2017

@Conan-Kudo @ignatenkobrain I have rebased this against master now. I checked that it builds, but did not run it against any spec files.

@ignatenkobrain

This comment has been minimized.

Copy link
Member

ignatenkobrain commented Aug 29, 2017

let me reopen PR to run CI tests

…uildTime()

Their use is causing difficult to diagnoze data races when building multiple
packages in parallel, and is a bad idea in general, as it also makes it more
difficult to reason about code.
@ignatenkobrain

This comment has been minimized.

Copy link
Member

ignatenkobrain commented Aug 29, 2017

@kanavin

spec.c: In function 'getBuildTime':
spec.c:214:9: error: 'errno' undeclared (first use in this function); did you mean 'h_errno'?
         errno = 0;
         ^~~~~
         h_errno
@kanavin kanavin force-pushed the kanavin:parallel-rpm branch from 8d9c63d to ae9f325 Aug 29, 2017
@kanavin

This comment has been minimized.

Copy link
Contributor Author

kanavin commented Aug 29, 2017

@ignatenkobrain Sorry, forgot to push the fix. Now fixed.

@ignatenkobrain

This comment has been minimized.

Copy link
Member

ignatenkobrain commented Aug 29, 2017

I will try to build texlive with and without PR and will report results :)

@ignatenkobrain

This comment has been minimized.

Copy link
Member

ignatenkobrain commented Aug 29, 2017

P.S. it has thousands of subpackages

@ignatenkobrain

This comment has been minimized.

Copy link
Member

ignatenkobrain commented Aug 29, 2017

Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/x86_64/hello-debugsource-1-1.fc28.x86_64.rpm

not sure if we should print those messages...

@ignatenkobrain

This comment has been minimized.

Copy link
Member

ignatenkobrain commented Aug 30, 2017

It would be nice to run dependency generators in parallel as well (although it might disallow us in future to do some advanced generators which would take subpackages into account, but AFAIK no one is planning to work on something like this in foreseeable future so doesn't seem to be a problem). This is what taking quite a lot of time for some of subpackages, so if they would run in parallel, this would speedup build.

Results

Before

real    56m45.083s                                  
user    39m34.072s                                  
sys     29m33.089s                                  

After

real    48m16.123s
user    51m54.226s
sys     31m9.800s

Hardware

$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              144
On-line CPU(s) list: 0-143
Thread(s) per core:  2
Core(s) per socket:  18
Socket(s):           4
NUMA node(s):        4
Vendor ID:           GenuineIntel
CPU family:          6
Model:               79
Model name:          Intel(R) Xeon(R) CPU E7-8860 v4 @ 2.20GHz
Stepping:            1
CPU MHz:             1200.305
CPU max MHz:         3200.0000
CPU min MHz:         1200.0000
BogoMIPS:            4389.14
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            46080K
NUMA node0 CPU(s):   0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128,132,136,140
NUMA node1 CPU(s):   1,5,9,13,17,21,25,29,33,37,41,45,49,53,57,61,65,69,73,77,81,85,89,93,97,101,105,109,113,117,121,125,129,133,137,141
NUMA node2 CPU(s):   2,6,10,14,18,22,26,30,34,38,42,46,50,54,58,62,66,70,74,78,82,86,90,94,98,102,106,110,114,118,122,126,130,134,138,142
NUMA node3 CPU(s):   3,7,11,15,19,23,27,31,35,39,43,47,51,55,59,63,67,71,75,79,83,87,91,95,99,103,107,111,115,119,123,127,131,135,139,143
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts

Comments

It didn't really seem to generate binary packages in parallel, but probably it worked, not sure ;)

Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-xespotcolor-svn40118-36.fc26.5.noarch.rpm                                                                                                                          
Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-xesearch-svn16041.0-36.fc26.5.noarch.rpm                                                          
Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-uothesis-doc-svn25355.2.5.6-36.fc26.5.noarch.rpm                                                                                                                   
Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-kdgdocs-doc-svn24498.1.0-36.fc26.5.noarch.rpm                                                     
Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-polyglossia-svn40138-36.fc26.5.noarch.rpm                                                                                                                          
Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-xespotcolor-svn40118-36.fc26.5.noarch.rpm              
Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-xesearch-svn16041.0-36.fc26.5.noarch.rpm                                                                                  
Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-uothesis-doc-svn25355.2.5.6-36.fc26.5.noarch.rpm   
Wrote: /home/brain/rpmbuild/RPMS/noarch/texlive-unicode-math-doc-svn38462-36.fc26.5.noarch.rpm                                                                                                                     
Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-unicode-math-doc-svn38462-36.fc26.5.noarch.rpm    
Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-kdgdocs-doc-svn24498.1.0-36.fc26.5.noarch.rpm                                                                             
Finished binary package job, result 0, filename /home/brain/rpmbuild/RPMS/noarch/texlive-polyglossia-svn40138-36.fc26.5.noarch.rpm              
@kanavin

This comment has been minimized.

Copy link
Contributor Author

kanavin commented Aug 30, 2017

@ignatenkobrain The reason we see a lot of speed up in Yocto with this patchset is that building the sources happens outside of rpm, and we only use rpm to do packaging of binaries. If the bulk of the above time is spend building texlive from source then of course the final difference won't be as much. Can you investigate a little bit where the time goes?

@ignatenkobrain

This comment has been minimized.

Copy link
Member

ignatenkobrain commented Aug 30, 2017

@kanavin, even 8 minutes speedup is good ;)

@proyvind

This comment has been minimized.

Copy link
Contributor

proyvind commented Feb 20, 2018

Shouldn't the time have come to review this again for finally merging this one after more than a half year now?

With rpm 4.14 out, it should be the right time to merge this one now, no? :)

@ffesti

This comment has been minimized.

Copy link
Contributor

ffesti commented Mar 5, 2018

Ok, I pushed the first and the last patch as a sign of good will.
The problem here is that librpm (which rpmstrPool is part of) but also librpmbuild are libraries that might be used in applications running other threading implementations.
Also rpm already uses pthread - although not very consistently. So yes, rpmstrPool needs some thread proofing. Still I am for now not comfortable to push these remaining patches without further looking into possible implications - especially on the librpm side.

@simotek

This comment has been minimized.

Copy link
Contributor

simotek commented Sep 6, 2018

@ignatenkobrain testing your patches in the Yocto project (with some extra debugging statements) on openSUSE's open build service instance its clear that the packages are still being built sequentially I'm not sure why yet, but here's some debug output below.

Some of us at SUSE are quite keen to get this working and I have time allocated to it primarily because it should speed up our kernel builds significantly but also because it will reduce the load on our build service significantly.

START:packageBinaries is printed at the start of the packageBinaries function, START:writerpm and END:writerpm are called either side of the writeRPM function the number being printed is just the address of the pkg pointer so I could keep track if the write's were actually being done in parallel [ 457s] is the timestamp of the current build in seconds

[  457s] ~~~~ START:packageSources ~~~~
[  457s] ~~~## START:writesourcerpm:37583280 ##~~~
[  457s] Wrote: /home/abuild/rpmbuild/SRPMS/efl-1.20.7-260.7.src.rpm
[  458s] ~~~## END:writesourcerpm:37583280 ##~~~
[  458s] ~~~~ END:packageSources ~~~~
[  458s] ~~~~ START:packageBinaries ~~~~
[  458s] ~~~## START:writerpm:37582688 ##~~~
[  458s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/efl-1.20.7-260.7.x86_64.rpm
[  470s] ~~~## END:writerpm:37582688 ##~~~
[  470s] ~~~## START:writerpm:37627248 ##~~~
[  470s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/efl-debugsource-1.20.7-260.7.x86_64.rpm
[  470s] ~~~## END:writerpm:37627248 ##~~~
[  470s] ~~~## START:writerpm:37630240 ##~~~
[  470s] Wrote: /home/abuild/rpmbuild/RPMS/noarch/efl-lang-1.20.7-260.7.noarch.rpm
[  475s] ~~~## END:writerpm:37630240 ##~~~
[  475s] ~~~## START:writerpm:37631968 ##~~~
[  475s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/efl-devel-1.20.7-260.7.x86_64.rpm
[  475s] ~~~## END:writerpm:37631968 ##~~~
[  475s] ~~~## START:writerpm:37642944 ##~~~
[  475s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libecore1-1.20.7-260.7.x86_64.rpm
[  475s] ~~~## END:writerpm:37642944 ##~~~
[  475s] ~~~## START:writerpm:37608864 ##~~~
[  475s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libector1-1.20.7-260.7.x86_64.rpm
[  476s] ~~~## END:writerpm:37608864 ##~~~
[  476s] ~~~## START:writerpm:37610224 ##~~~
[  476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libedje1-1.20.7-260.7.x86_64.rpm
[  476s] ~~~## END:writerpm:37610224 ##~~~
[  476s] ~~~## START:writerpm:37612560 ##~~~
[  476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeldbus1-1.20.7-260.7.x86_64.rpm
[  476s] ~~~## END:writerpm:37612560 ##~~~
[  476s] ~~~## START:writerpm:37614288 ##~~~
[  476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeet1-1.20.7-260.7.x86_64.rpm
[  476s] ~~~## END:writerpm:37614288 ##~~~
[  476s] ~~~## START:writerpm:37665200 ##~~~
[  476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeeze1-1.20.7-260.7.x86_64.rpm
[  476s] ~~~## END:writerpm:37665200 ##~~~
[  476s] ~~~## START:writerpm:37666688 ##~~~
[  476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefl1-1.20.7-260.7.x86_64.rpm
[  476s] ~~~## END:writerpm:37666688 ##~~~
[  476s] ~~~## START:writerpm:37669168 ##~~~
[  476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefreet1-1.20.7-260.7.x86_64.rpm
[  476s] ~~~## END:writerpm:37669168 ##~~~
[  476s] ~~~## START:writerpm:37671472 ##~~~
[  476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefreet_mime1-1.20.7-260.7.x86_64.rpm
[  476s] ~~~## END:writerpm:37671472 ##~~~
[  476s] ~~~## START:writerpm:37774416 ##~~~
[  476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefreet_trash1-1.20.7-260.7.x86_64.rpm
[  476s] ~~~## END:writerpm:37774416 ##~~~
[  476s] ~~~## START:writerpm:37774000 ##~~~
[  476s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeina1-1.20.7-260.7.x86_64.rpm
[  477s] ~~~## END:writerpm:37774000 ##~~~
[  477s] ~~~## START:writerpm:37775984 ##~~~
[  477s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeio1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37775984 ##~~~
[  478s] ~~~## START:writerpm:37777648 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelementary1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37777648 ##~~~
[  478s] ~~~## START:writerpm:37779280 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelocation1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37779280 ##~~~
[  478s] ~~~## START:writerpm:37781248 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelput1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37781248 ##~~~
[  478s] ~~~## START:writerpm:37782784 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelua1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37782784 ##~~~
[  478s] ~~~## START:writerpm:37876144 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libembryo1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37876144 ##~~~
[  478s] ~~~## START:writerpm:37877696 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libemotion1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37877696 ##~~~
[  478s] ~~~## START:writerpm:37879984 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libemile1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37879984 ##~~~
[  478s] ~~~## START:writerpm:37881376 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeo1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37881376 ##~~~
[  478s] ~~~## START:writerpm:37833856 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeolian1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37833856 ##~~~
[  478s] ~~~## START:writerpm:37835344 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libethumb1-1.20.7-260.7.x86_64.rpm
[  478s] ~~~## END:writerpm:37835344 ##~~~
[  478s] ~~~## START:writerpm:37837008 ##~~~
[  478s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libephysics1-1.20.7-260.7.x86_64.rpm
[  479s] ~~~## END:writerpm:37837008 ##~~~
[  479s] ~~~## START:writerpm:37838960 ##~~~
[  479s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libevas1-1.20.7-260.7.x86_64.rpm
[  479s] ~~~## END:writerpm:37838960 ##~~~
[  479s] ~~~## START:writerpm:37840496 ##~~~
[  479s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libethumb_client1-1.20.7-260.7.x86_64.rpm
[  480s] ~~~## END:writerpm:37840496 ##~~~
[  480s] ~~~## START:writerpm:37801952 ##~~~
[  480s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/edje-1.20.7-260.7.x86_64.rpm
[  484s] ~~~## END:writerpm:37801952 ##~~~
[  484s] ~~~## START:writerpm:37804000 ##~~~
[  484s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/elementary-1.20.7-260.7.x86_64.rpm
[  484s] ~~~## END:writerpm:37804000 ##~~~
[  484s] ~~~## START:writerpm:37807440 ##~~~
[  484s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/elementary-examples-1.20.7-260.7.x86_64.rpm
[  484s] ~~~## END:writerpm:37807440 ##~~~
[  484s] ~~~## START:writerpm:37924816 ##~~~
[  484s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/elua-1.20.7-260.7.x86_64.rpm
[  484s] ~~~## END:writerpm:37924816 ##~~~
[  484s] ~~~## START:writerpm:37926784 ##~~~
[  484s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/embryo-1.20.7-260.7.x86_64.rpm
[  484s] ~~~## END:writerpm:37926784 ##~~~
[  484s] ~~~## START:writerpm:37928752 ##~~~
[  484s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/evas-generic-loaders-1.20.7-260.7.x86_64.rpm
[  487s] ~~~## END:writerpm:37928752 ##~~~
[  487s] ~~~## START:writerpm:37930992 ##~~~
[  487s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/efl-examples-1.20.7-260.7.x86_64.rpm
[  487s] ~~~## END:writerpm:37930992 ##~~~
[  487s] ~~~## START:writerpm:37932608 ##~~~
[  487s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/efl-testsuite-1.20.7-260.7.x86_64.rpm
[  490s] ~~~## END:writerpm:37932608 ##~~~
[  490s] ~~~## START:writerpm:37809472 ##~~~
[  490s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/enlightenment-theme-upstream-0.21.0-260.7.x86_64.rpm
[  493s] ~~~## END:writerpm:37809472 ##~~~
[  493s] ~~~## START:writerpm:37811968 ##~~~
[  493s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/enlightenment-theme-dark-0.21.0-260.7.x86_64.rpm
[  493s] ~~~## END:writerpm:37811968 ##~~~
[  493s] ~~~## START:writerpm:37814384 ##~~~
[  493s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/enlightenment-x-dark-icon-theme-0.21.0-260.7.x86_64.rpm
[  499s] ~~~## END:writerpm:37814384 ##~~~
[  499s] ~~~## START:writerpm:37625936 ##~~~
[  499s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/efl-debuginfo-1.20.7-260.7.x86_64.rpm
[  501s] ~~~## END:writerpm:37625936 ##~~~
[  501s] ~~~## START:writerpm:43870128 ##~~~
[  501s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/efl-devel-debuginfo-1.20.7-260.7.x86_64.rpm
[  503s] ~~~## END:writerpm:43870128 ##~~~
[  503s] ~~~## START:writerpm:41441408 ##~~~
[  503s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libecore1-debuginfo-1.20.7-260.7.x86_64.rpm
[  503s] ~~~## END:writerpm:41441408 ##~~~
[  503s] ~~~## START:writerpm:42232032 ##~~~
[  503s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libector1-debuginfo-1.20.7-260.7.x86_64.rpm
[  505s] ~~~## END:writerpm:42232032 ##~~~
[  505s] ~~~## START:writerpm:44036368 ##~~~
[  505s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libedje1-debuginfo-1.20.7-260.7.x86_64.rpm
[  505s] ~~~## END:writerpm:44036368 ##~~~
[  505s] ~~~## START:writerpm:42311248 ##~~~
[  505s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeldbus1-debuginfo-1.20.7-260.7.x86_64.rpm
[  505s] ~~~## END:writerpm:42311248 ##~~~
[  505s] ~~~## START:writerpm:42311712 ##~~~
[  505s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeet1-debuginfo-1.20.7-260.7.x86_64.rpm
[  505s] ~~~## END:writerpm:42311712 ##~~~
[  505s] ~~~## START:writerpm:44183296 ##~~~
[  505s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeeze1-debuginfo-1.20.7-260.7.x86_64.rpm
[  506s] ~~~## END:writerpm:44183296 ##~~~
[  506s] ~~~## START:writerpm:41396416 ##~~~
[  506s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefl1-debuginfo-1.20.7-260.7.x86_64.rpm
[  506s] ~~~## END:writerpm:41396416 ##~~~
[  506s] ~~~## START:writerpm:40907472 ##~~~
[  506s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefreet1-debuginfo-1.20.7-260.7.x86_64.rpm
[  506s] ~~~## END:writerpm:40907472 ##~~~
[  506s] ~~~## START:writerpm:41391808 ##~~~
[  506s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefreet_mime1-debuginfo-1.20.7-260.7.x86_64.rpm
[  506s] ~~~## END:writerpm:41391808 ##~~~
[  506s] ~~~## START:writerpm:43229904 ##~~~
[  506s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libefreet_trash1-debuginfo-1.20.7-260.7.x86_64.rpm
[  507s] ~~~## END:writerpm:43229904 ##~~~
[  507s] ~~~## START:writerpm:42711120 ##~~~
[  507s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeina1-debuginfo-1.20.7-260.7.x86_64.rpm
[  507s] ~~~## END:writerpm:42711120 ##~~~
[  507s] ~~~## START:writerpm:41072128 ##~~~
[  507s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeio1-debuginfo-1.20.7-260.7.x86_64.rpm
[  512s] ~~~## END:writerpm:41072128 ##~~~
[  512s] ~~~## START:writerpm:37989728 ##~~~
[  512s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelementary1-debuginfo-1.20.7-260.7.x86_64.rpm
[  512s] ~~~## END:writerpm:37989728 ##~~~
[  512s] ~~~## START:writerpm:41288480 ##~~~
[  512s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelocation1-debuginfo-1.20.7-260.7.x86_64.rpm
[  512s] ~~~## END:writerpm:41288480 ##~~~
[  512s] ~~~## START:writerpm:43342752 ##~~~
[  512s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelput1-debuginfo-1.20.7-260.7.x86_64.rpm
[  512s] ~~~## END:writerpm:43342752 ##~~~
[  512s] ~~~## START:writerpm:43124208 ##~~~
[  512s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libelua1-debuginfo-1.20.7-260.7.x86_64.rpm
[  512s] ~~~## END:writerpm:43124208 ##~~~
[  512s] ~~~## START:writerpm:37924192 ##~~~
[  512s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libembryo1-debuginfo-1.20.7-260.7.x86_64.rpm
[  512s] ~~~## END:writerpm:37924192 ##~~~
[  512s] ~~~## START:writerpm:44740400 ##~~~
[  512s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libemotion1-debuginfo-1.20.7-260.7.x86_64.rpm
[  513s] ~~~## END:writerpm:44740400 ##~~~
[  513s] ~~~## START:writerpm:41465376 ##~~~
[  513s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libemile1-debuginfo-1.20.7-260.7.x86_64.rpm
[  513s] ~~~## END:writerpm:41465376 ##~~~
[  513s] ~~~## START:writerpm:38012256 ##~~~
[  513s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeo1-debuginfo-1.20.7-260.7.x86_64.rpm
[  513s] ~~~## END:writerpm:38012256 ##~~~
[  513s] ~~~## START:writerpm:40414736 ##~~~
[  513s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libeolian1-debuginfo-1.20.7-260.7.x86_64.rpm
[  513s] ~~~## END:writerpm:40414736 ##~~~
[  513s] ~~~## START:writerpm:42748160 ##~~~
[  513s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libethumb1-debuginfo-1.20.7-260.7.x86_64.rpm
[  514s] ~~~## END:writerpm:42748160 ##~~~
[  514s] ~~~## START:writerpm:43754176 ##~~~
[  514s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libephysics1-debuginfo-1.20.7-260.7.x86_64.rpm
[  518s] ~~~## END:writerpm:43754176 ##~~~
[  518s] ~~~## START:writerpm:43852032 ##~~~
[  518s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libevas1-debuginfo-1.20.7-260.7.x86_64.rpm
[  518s] ~~~## END:writerpm:43852032 ##~~~
[  518s] ~~~## START:writerpm:47427392 ##~~~
[  518s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/libethumb_client1-debuginfo-1.20.7-260.7.x86_64.rpm
[  519s] ~~~## END:writerpm:47427392 ##~~~
[  519s] ~~~## START:writerpm:43164560 ##~~~
[  519s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/edje-debuginfo-1.20.7-260.7.x86_64.rpm
[  519s] ~~~## END:writerpm:43164560 ##~~~
[  519s] ~~~## START:writerpm:42696096 ##~~~
[  519s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/elementary-debuginfo-1.20.7-260.7.x86_64.rpm
[  519s] ~~~## END:writerpm:42696096 ##~~~
[  519s] ~~~## START:writerpm:47993152 ##~~~
[  519s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/embryo-debuginfo-1.20.7-260.7.x86_64.rpm
[  519s] ~~~## END:writerpm:47993152 ##~~~
[  519s] ~~~## START:writerpm:46933104 ##~~~
[  519s] Wrote: /home/abuild/rpmbuild/RPMS/x86_64/evas-generic-loaders-debuginfo-1.20.7-260.7.x86_64.rpm
[  519s] ~~~## END:writerpm:46933104 ##~~~
[  519s] ~~~~ END:packageBinaries ~~~~
@simotek

This comment has been minimized.

Copy link
Contributor

simotek commented Sep 6, 2018

Never mind it seems that the patch is not being applied correctly here

@simotek

This comment has been minimized.

Copy link
Contributor

simotek commented Sep 13, 2018

Is it possible to get this accepted? I've been testing it on openSUSE's build services, with some very rough (semi controlled) benchmarking the speedup's aren't huge a medium size package that splits out into multiple packages might save 10 seconds off a 5-6 minute build but when you take that across a full distro rebuild of 10,000+ packages it will be significant.

Then there are things like libreoffice which is a silly long build, where the building the binary process used to take 900 seconds and is now cut down to 125 seconds, which equates to around an 8 minutes quicker although in the context of a almost 4 hr build time.

I am now looking into making a similar change for the processing binaries stage.

pmatilai added a commit that referenced this pull request Sep 28, 2018
The shared string pool is in a very central role in several operations,
it's kinda embarrasing that we haven't had any thread protection
on it. Not that anybody has asked either, prior to coming up as part
of #226 (to enable threaded package creation).

Test-suite and couple of smoke-tests with the #226 pass, but only
lightly tested. Then again, it's relatively straightforward. As a general
rule, locks are taken on all exported interfaces on entry and released
on exit, internal callers never lock anything. In rpm usage at least,
performance hit seems negligible.
@pmatilai

This comment has been minimized.

Copy link
Contributor

pmatilai commented Sep 28, 2018

FWIW, thread protection added to the string pool as of commit 7ffc4d1 so that part of these patches can be dropped. Brief testing showed no problems with the rest applied, but I don't have any real testcase for parallelism at hand.

pmatilai added a commit to pmatilai/rpm that referenced this pull request May 7, 2019
It's worth noting that this really is walking on thin ice as only few
parts of rpm are thread-protected. The spec is entirely unprotected so
must be accessed only for read-only purposes from parallel jobs (and
we should work towards enforcing that via const-correctness and other
protection as needed), and similarly the package struct and headers
are unprotected so they can only be manipulated within a single thread.

Based on initial work by Alexander Kanavin in PR rpm-software-management#226
@pmatilai

This comment has been minimized.

Copy link
Contributor

pmatilai commented May 8, 2019

For interested parties, I submitted a more elaborate version as #695 with further code cleanups and additional parallelization of file classifier.

@kanavin

This comment has been minimized.

Copy link
Contributor Author

kanavin commented May 8, 2019

@pmatilai I trust that the last two commits in my set (make string pool operations thread-safe; avoid use of static data) are already merged into master in some form? Don't see those fixes in #695.

@pmatilai

This comment has been minimized.

Copy link
Contributor

pmatilai commented May 9, 2019

Yes, as already mentioned in an earlier comment to this PR: #226 (comment)

pmatilai added a commit to pmatilai/rpm that referenced this pull request May 14, 2019
It's worth noting that this really is walking on thin ice as only few
parts of rpm are thread-protected. The spec is entirely unprotected so
must be accessed only for read-only purposes from parallel jobs (and
we should work towards enforcing that via const-correctness and other
protection as needed), and similarly the package struct and headers
are unprotected so they can only be manipulated within a single thread.

Based on initial work by Alexander Kanavin in PR rpm-software-management#226
pmatilai added a commit to pmatilai/rpm that referenced this pull request May 14, 2019
It's worth noting that this really is walking on thin ice as only few
parts of rpm are thread-protected. The spec is entirely unprotected so
must be accessed only for read-only purposes from parallel jobs (and
we should work towards enforcing that via const-correctness and other
protection as needed), and similarly the package struct and headers
are unprotected so they can only be manipulated within a single thread.

Based on initial work by Alexander Kanavin in PR rpm-software-management#226
@ffesti

This comment has been minimized.

Copy link
Contributor

ffesti commented May 16, 2019

As #695 supersedes this I am closing this PRs.
Thanks for your work and pushing this issue forwards. Those PRs clearly wouldn't be there if it wasn't for you!

@ffesti ffesti closed this May 16, 2019
ffesti added a commit that referenced this pull request May 21, 2019
It's worth noting that this really is walking on thin ice as only few
parts of rpm are thread-protected. The spec is entirely unprotected so
must be accessed only for read-only purposes from parallel jobs (and
we should work towards enforcing that via const-correctness and other
protection as needed), and similarly the package struct and headers
are unprotected so they can only be manipulated within a single thread.

Based on initial work by Alexander Kanavin in PR #226
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.