Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[review] Paranoid mode for `ldd -r` #6

Closed
wants to merge 25 commits into from
Closed

Conversation

@bartoszek
Copy link
Contributor

bartoszek commented Nov 25, 2019

ldd -r determine possible issues with shared object resolution.
scanelf -pls resolve issues from ldd, and report real issues.
This is just a sketch, it need performance improvement by caching scanelf -pls output.
Fix #5

@bartoszek bartoszek changed the title Sloooooooow fix for `ldd -r` using `scanelf` [wip] Sloooooooow fix for `ldd -r` using `scanelf` Nov 25, 2019
@maximbaz

This comment has been minimized.

Copy link
Owner

maximbaz commented Nov 25, 2019

Very cool! Hopefully we can make this performant, will keep an eye on your progress 👍

@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Nov 25, 2019

In the worst case we can make it a --paranoid option. And normally just limit ldd -r to $PATH and ld.config...

@bartoszek bartoszek force-pushed the bartoszek:scanelf branch from b2620bd to de8e257 Nov 26, 2019
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Nov 26, 2019

~5x speedup, still to slow for pacman hook

Main problem are packages encapsulating it's own python, resulting in hundreds of unresolved symbols ( e.g. paraview-opt with ~300 python modules each having ~200+ unresolved symbols)

$ pacman -Qql paraview-opt|xargs -I{} sh -c "file \"{}\"|grep -q ELF && echo {}"| xargs ldd -r|grep -oP "undefined sy
mbol: \K.*(?=\t)"|sort|uniq -c|sort -n|nl
     1        1 dladdr                                                               
     2        1 pthread_create                                                       
     3        1 pthread_getspecific 
     4        1 pthread_key_create                                                   
     5        1 pthread_once           
     6        1 pthread_setspecific    
     7        1 PyThreadState_Get 
     8        2 PyCapsule_GetName        
     9        2 PyCapsule_GetPointer
    10        2 PyCapsule_IsValid                                                    
    11        2 PyErr_WarnEx        
    12        2 PyExc_IndexError          
    13        2 PyOS_snprintf    
    14        2 PyUnicode_EncodeLocale
    15        6 _Py_NotImplementedStruct 
    16        8 PyDict_Keys      
    17        8 Py_GetPlatform 
...
   187      354 PyUnicode_AsUTF8  
   188      358 PyObject_GetAttrString
   189      377 _Py_NoneStruct
   190      396 PyLong_FromLong  
   191      405 PyType_Ready      
   192      405 PyType_Type    
   193      407 PyModule_GetDict
   194      412 PyImport_ImportModule
   195      435 _Py_Dealloc
   196      461 PyErr_Print  
   197      470 PyUnicode_FromString   
   198      492 PyObject_Free    
   199      537 PyErr_Occurred   
   200      592 PyErr_Clear    
   201      598 PyObject_GenericGetAttr

Searching throuhg those 201 symbols took ~2m30s for single scanelf -pls call.

@bartoszek bartoszek force-pushed the bartoszek:scanelf branch 4 times, most recently from 4177e0c to c75e9eb Nov 26, 2019
@bartoszek bartoszek changed the title [wip] Sloooooooow fix for `ldd -r` using `scanelf` [testing] Paranoid mode for `ldd -r` Nov 27, 2019
@bartoszek bartoszek force-pushed the bartoszek:scanelf branch 2 times, most recently from 9b2e217 to 4055476 Nov 27, 2019
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Nov 27, 2019

@maximbaz You can start testing, it's slow as $@#%@ but should resolves everything on your system 😑 please report if it didn't.

@maximbaz

This comment has been minimized.

Copy link
Owner

maximbaz commented Nov 27, 2019

Woow, that's quite a PR!

First of all, thank you for this, I really appreciate.

I tried it out, it is truly an impressive piece of work, and UI is fun too, with progress bar and everything 🙂

I have a big honest question though, given that it has to become this slow to give a 100% correct report, do you think it is useful?

I'm running this while typing this text, so maybe for around 5 minutes, and ETA currently says I need to wait for 21 more minutes, while CPU usage is at 100%, and I have a decent 8-core Intel i7-7820HQ and not that many packages installed...

@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Nov 28, 2019

Think it can be ~10x faster utilizing shared memory, even faster using IPC in python or rust.

As for usefulness, that's for community to decide. I'm sourly going to use it, perhaps as a Cron job if can't speed this up to a pacman hook level.

Still need some work, with better caching and shared object in optional dependency resolution.
Also need to test a backwards symbol resolution.

@maximbaz

This comment has been minimized.

Copy link
Owner

maximbaz commented Nov 28, 2019

I asked around a bit, a unanimous feedback was that a tool that can detect 90% of issues in 1 second is significantly more valuable than a tool that can detect 100% of issues in 1 minute, let alone 30 minutes. And to be honest I share this view.

I definitely don't want to discourage you, but maybe it is worth considering to create a new project and implement everything from scratch (e.g. in rust as you said), and not be bound by limitations of this project (e.g. it being written in bash).

In the worst case we can make it a --paranoid option. And normally just limit ldd -r to $PATH and ld.config...

By the way, I tried this approach out, it doesn't seem to be fail-proof. For example, dell-command-configure installs a file /opt/dell/dcc/libhapiintf.so, its folder is present in ldconfig -vN 2>/dev/null | grep -v ^$'\t', but at the same time ldd -r finds some issues on it that I can't fix by rebuild.

$ ldd -r /opt/dell/dcc/libhapiintf.so
	linux-vdso.so.1 (0x00007fffabbf3000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f64b462b000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f64b4611000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f64b444a000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f64b4304000)
	/usr/lib64/ld-linux-x86-64.so.2 (0x00007f64b4b20000)
undefined symbol: DCHBASCallingInterfaceCommandEx	(/opt/dell/dcc/libhapiintf.so)
undefined symbol: DCHBASCallingInterfaceCommand	(/opt/dell/dcc/libhapiintf.so)
undefined symbol: DCHBASSMBIOSCommand	(/opt/dell/dcc/libhapiintf.so)
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Nov 28, 2019

tool that can detect 90% of issues in 1 second is significantly more valuable than a tool that can detect 100% of issues in 1 minute

I'm sharing this view 100%, that's why I've made a separate branch that only probes $PATH binaries with ldd -r and omits libraries all together (I consider system fine, if there is a broken library but there's no binary that depend on it).

This is by fare the fastest route, and it's sufficient for me as pacman.hook.

@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Nov 28, 2019

Just realized my infinite stupidity 😵 : why caching resolved_symbols and not unresolved_symbols 😲 it would bring ~10x speed up.

@bartoszek bartoszek force-pushed the bartoszek:scanelf branch from 4055476 to ad2d066 Nov 28, 2019
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Nov 28, 2019

@maximbaz got under 10min with ~3500 packages. Everything resolves - beside hplip which I've reported at arch-bugtracker 😏
Most of the time it spend in resolving ~500 (python|perl|haskell)-xxx packages this could be speed up significantly if done in series instead parallel.

@bartoszek bartoszek force-pushed the bartoszek:scanelf branch 2 times, most recently from ace2f1d to 49860ac Nov 28, 2019
@bartoszek bartoszek changed the title [testing] Paranoid mode for `ldd -r` [review] Paranoid mode for `ldd -r` Nov 29, 2019
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Nov 29, 2019

@maximbaz got under 15sec with foreign+ packages, please review - than I'll clean it for the merge.

  • reports timing with -v flag
  • Still to slow for system wide search 8<>15min
@maximbaz

This comment has been minimized.

Copy link
Owner

maximbaz commented Nov 29, 2019

I have no foreign packages, I use aurutils and all my AUR packages are hosted in a special repo (that's the reason for the current implementation of get_unofficial_pkgs).

If I revert to the original implementation of get_unofficial_pkgs, the performance is 5.5 seconds on my machine! Which is very, very nice!

@maximbaz

This comment has been minimized.

Copy link
Owner

maximbaz commented Nov 29, 2019

I haven't looked at the code yet, I tried to execute it via pacman hook to see if 5.5s will feel painful, and it seems something gets stuck, it hangs and keeps using CPU forever, I had to cancel. If you have time, could you please try if you reproduce?

@bartoszek bartoszek force-pushed the bartoszek:scanelf branch from 49860ac to 6a6d892 Nov 29, 2019
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Nov 29, 2019

Should be fixed with bf9ce43 somehow eval parallel ... produced bugged pacman hook.

@bartoszek bartoszek force-pushed the bartoszek:scanelf branch from a7f3aaa to 5cfcd1a Nov 29, 2019
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Nov 29, 2019

I've changed flags:

  • -v: show ldd logs.
  • -vv: show jobs statistics at the end.
  • -s: resolve all installed packages.
bartus added 7 commits Nov 26, 2019
There're a horde of files that has executable bit set
but has noting to do with ELFs (png, html, js, etc.)
Required for `scanelf` caching across entire package.
This allows to check if package contains missing
shared object before declaring it as unresolved.
* Resolve missing shared objects inaccessible
  by `ld.so` but available in the package.
* Update `ldd -r`.
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Dec 4, 2019

I'm pretty much done.
Stuff left:

  • cache broken packages in /var/log/rebuild-detector/broken-pkgs and make hook list them until they got fixed.
  • refactor get_unofficial_packages to fulfill all combination of flags and stdin.
@bartoszek bartoszek force-pushed the bartoszek:scanelf branch from 6f2a20e to 9ce21a5 Dec 4, 2019
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Dec 4, 2019

I've added NeedsTargets to rebuild-detectro.hook to limit the scope of checkrebuild. It utilizes pactree reverse dependency feature to list packages that depends one upgraded ones. You can change depth of dependency tree with -d x flag (0: no dependency, 1: immediate ones, etc.)

With this feature, hook runs pretty much instantaneously 😏

@bartoszek bartoszek force-pushed the bartoszek:scanelf branch from 9ce21a5 to 07e11a0 Dec 4, 2019
@maximbaz

This comment has been minimized.

Copy link
Owner

maximbaz commented Dec 4, 2019

NeedsTargets is a very interesting idea 👍

cache broken packages in /var/log/rebuild-detector/broken-pkgs and make hook list them until they got fixed.

What do you mean by this, a run of checkrebuild will create this file, and some other hook (pacman hook?) will just look at the presence of that file?

bartus added 13 commits Nov 27, 2019
How it works:
1. Create dictionary of symbols resolved with `scanelf`.
2. Probe `undefined_symbls` with `ldd -r`.
3. Drop resolved symbols already in cache.
3. Scan package files for left `undefined_symbol`.
4. Add resolved symbols to cache.
5. Scan system for left `undefined_symbols`
6. Add resolved symbols to cache.
7. If anything left, report package as broken.
Split second stage of `scanelf` to `ld.conf` and `$PATH`
and a brute force recursive deep search.

Encapsulate `resolve_symbol_cache`
* add multiple verbose level (-v|-vv)
  * verbose=1 show ldd logs.
  * verbose=2 show parallel jobs statistics
@bartoszek bartoszek force-pushed the bartoszek:scanelf branch from 07e11a0 to 6e0e058 Dec 5, 2019
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Dec 5, 2019

I've optimized the shared cache a bit, shaved of ~120 sec. from previous implementation (303 sec. in runtime: perl-wx goes down by ~3 min. 😊 )

bartus@grafZero:~/AUR/rebuild-detector [1710] <:3 )~ ./time.checkrebuild
Execute: checkrebuild
100% 488:0=0s zpaq
0.01user 0.00system 4:15.70elapsed 0%CPU (0avgtext+0avgdata 4328maxresident)k
0inputs+8outputs (0major+1177minor)pagefaults 0swaps
Execute: 0cb3b57
100% 488:0=0s zpaq
0.01user 0.00system 6:08.05elapsed 0%CPU (0avgtext+0avgdata 4380maxresident)k
0inputs+8outputs (0major+1201minor)pagefaults 0swaps
checkrebuild                          0cb3b57
aur     amide-hg                      aur     amide-hg
aur     awesomebump                   aur     awesomebump
aur     blender-plugin-animation-nodesaur     blender-plugin-animation-nodes
aur     boost-63-compat               aur     boost-63-compat
aur     libg3d                        aur     libg3d
aur     liblas                        aur     liblas
aur     luxrender-hg                  aur     luxrender-hg
aur     python-nvidia-ml-py3-git      aur     python-nvidia-ml-py3-git
aur     radare2-bindings-git          aur     radare2-bindings-git
aur     teigha-file-converter         aur     teigha-file-converter
aur     teighaviewer-qt5              aur     teighaviewer-qt5
aur     vtk6                          aur     vtk6
aur     vxl                           aur     vxl
aur     xnviewmp                      aur     xnviewmp
foreign cinelerra-cv                  foreign cinelerra-cv
foreign cppqed-git                    foreign cppqed-git
foreign kdelibs                       foreign kdelibs

test runs:
A:      checkrebuild
B:      0cb3b57
comparisons:                                    A       A-B

                      dockbarx-deep-theme       0.022   -0.004
                             gnome-python       0.023   -0.002
                           python2-ipaddr       0.023   -0.003
                    dockbarx-macosx-theme       0.023   0
                      dockbarx-dock-theme       0.024   -0.005
                                   bacpac       0.029   0.001
                              gcc-gcj-ecj       0.029   -0.001
                          python2-notify2       0.029   -0.003
                  linux-pf-preset-default       0.029   0
                dockbarx-glassified-theme       0.030   0.004
                              ttf-symbola       0.031   -0.011
                 python-nvidia-ml-py3-git       0.031   -0.001
             blender-plugin-uvsquares-git       0.031   -0.006
             perl-xml-simpleobject-libxml       0.031   -0.005
                                  fakepkg       0.032   0.002
                            python2-polib       0.032   -0.002
                        nvidia-ck-haswell       0.032   0
                             aur-talk-git       0.033   -0.006
                             ttf-ubraille       0.033   0
                         customizepkg-git       0.033   -0.001
                                      nvm       0.034   -0.004
                   python2-qscintilla-qt4       0.034   0.002
                                drive-bin       0.035   -0.014
                               ccache-ext       0.035   -0.011
                         perl-test-tester       0.035   -0.012
                  dockbarx-shinybar-theme       0.035   0.002
                blender-plugin-cubesurfer       0.035   -0.004
                            ttf-zekton-rg       0.036   0
                        ttf-mph-2b-damase       0.036   0
                     dropbox-uploader-git       0.036   -0.005
                     perl-math-convexhull       0.036   0.001
                    vivaldi-ffmpeg-codecs       0.036   0.013
                  blender-plugin-mesh-off       0.036   0.004
                 blender-plugin-molecular       0.036   -0.001
               perl-module-build-withxspp       0.036   -0.010
                         python2-zeroconf       0.037   0
                      perl-constant-defer       0.037   -0.005
                                  profont       0.038   0.006
                               imgurbash2       0.038   0.004
                              dropbox-cli       0.038   -0.009
                                      bfg       0.039   0.005
                                     lpaq       0.039   -0.011
                                openstego       0.039   0.005
                             haskell-nats       0.039   0
                            torrent-mount       0.039   0.008
                           python2-docopt       0.039   0
                         makepkg-optimize       0.039   -0.007
                         rebuild-detector       0.039   0.003
              blender-plugin-gcode-reader       0.039   0.007
                                   iwatch       0.040   -0.012
                        chromium-widevine       0.040   0.003
                         radare2-pipe-git       0.041   0.006
                                bashmount       0.042   0
                               gdrive-git       0.042   -0.004
                              glew-1.13.0       0.042   0.004
                      blender-plugin-blam       0.042   0.010
                                    paq8o       0.043   -0.003
                   universal-gcode-sender       0.043   -0.024
                   kaitai-struct-compiler       0.044   -0.003
                       python2-repoze.lru       0.045   -0.004
                              vim-systemd       0.046   0.016
             android-backup-extractor-git       0.046   0.001
                              icon-slicer       0.047   -0.012
                         lib32-libstdc++5       0.047   -0.007
                             python2-ipdb       0.048   -0.008
                           python2-routes       0.048   -0.013
                          hardening-check       0.048   0.002
                        package-query-git       0.048   -0.010
              transmission-remote-cli-git       0.048   0.004
                                    sbsms       0.049   -0.006
                                  archey4       0.049   0.003
                                javahelp2       0.049   -0.001
                                mp4fpsmod       0.049   -0.016
                          python-casttube       0.049   0.003
              blender-plugin-blendrig-git       0.049   -0.003
                                   levmar       0.050   -0.005
                                 meshconv       0.050   0
                             sndpeek-alsa       0.050   -0.005
                     clean-chroot-manager       0.051   0.012
                                     balz       0.052   0.001
                                   pngout       0.052   -0.002
                                  udefrag       0.052   0.010
                                 libpng15       0.052   -0.003
                              qtcurve-qt4       0.052   -0.007
                            python2-gconf       0.053   -0.008
                                     cava       0.054   0.004
                                    scrub       0.055   0.005
                                  librelp       0.055   -0.006
                             archivemount       0.055   0.003
                           ttf-cheapskate       0.055   -0.001
                                    oneko       0.056   0.004
                                expac-git       0.056   0.004
                              gcode3d-git       0.056   0.006
                     anything-sync-daemon       0.056   -0.009
                                     gede       0.057   -0.006
                                   qt4pas       0.057   -0.003
                    python2-gtkspellcheck       0.057   -0.008
                             python2-yenc       0.058   0.003
                      blender-plugin-cork       0.058   -0.004
                                   winpdb       0.059   0.009
                                  3mf2stl       0.059   0.012
                                  fatrace       0.059   0.002
                           qextserialport       0.059   -0.003
                                   xcalib       0.060   -0.001
                                  libestr       0.060   0.007
                                  siftgpu       0.060   -0.013
                                     ccat       0.061   0.019
                                   dcadec       0.061   -0.002
                                    mtree       0.062   0.010
                                 yagv-git       0.062   0.020
                               medleytext       0.062   0.013
                                 spnavcfg       0.063   -0.002
                                hg-git-hg       0.063   0.001
                               downgrader       0.063   0.008
                             ddrescueview       0.063   -0.002
                             perl-rpc-xml       0.063   -0.003
                                stl-thumb       0.064   0.012
                                swapspace       0.064   0.011
                               repose-git       0.064   0.009
                blender-plugin-gaffer-git       0.064   0.009
           perl-extutils-typemaps-default       0.064   -0.014
                                     gksu       0.065   0.002
                                 automoc4       0.065   0.006
                                 cork-git       0.065   -0.006
                              libfastjson       0.065   -0.003
                             qt5-canvas3d       0.065   0.003
                            python2-pyusb       0.065   -0.025
                                 verynice       0.066   0.004
                            skippy-xd-git       0.066   0.007
          blender-plugin-light-studio-git       0.066   -0.011
                               urlgrabber       0.067   -0.015
                                zgrviewer       0.068   0.013
                              untrunc-git       0.068   0.003
                             uvesafb-dkms       0.068   -0.003
                            bcompare-kde5       0.068   -0.014
                                   w_scan       0.069   0.010
                                hpmvs-git       0.069   -0.014
                          logmein-hamachi       0.069   -0.003
                          openblas-lapack       0.069   -0.002
blender-plugin-bullet-constraints-builder       0.070   -0.015
                              libechonest       0.071   -0.021
                            pcompress-git       0.071   -0.016
                    v4l2loopback-dkms-git       0.071   -0.016
                                    lglaf       0.072   -0.013
                               glc-player       0.072   0.004
                           python2-pydbus       0.072   -0.007
                        python2-terminado       0.072   0.026
                     embree-bvh_build-git       0.072   -0.009
                                    agedu       0.073   0.006
                                   lf-git       0.074   0.003
                                  yay-git       0.074   0.006
                     blender-plugin-yavne       0.075   0.003
                    perl-extutils-parsexs       0.075   0
                         python-numpy-stl       0.076   0.007
                                     mpir       0.078   -0.003
                                   signal       0.078   0.014
                       ttf-freebanglafont       0.078   -0.001
                                diffmerge       0.079   0.013
                               liblogging       0.080   -0.016
                               spectrum3d       0.080   0.007
                            python2-txaio       0.080   -0.002
                                libnm-gtk       0.081   0.003
                                    qjson       0.082   0.022
                                  gconfmm       0.082   0.003
                              auracle-git       0.082   0.005
                             artwiz-fonts       0.082   -0.013
                                  nocache       0.083   -0.002
                                     zpaq       0.085   0.027
                            python2-pyqt4       0.086   -0.012
                                 btfs-git       0.088   0.006
                                 stlsplit       0.088   0.015
                            termsaver-git       0.088   -0.079
                    xfce4-dockbarx-plugin       0.088   0.013
                              telegramqml       0.090   -0.015
                           green-recorder       0.090   -0.017
                   ruby-net-http-pipeline       0.091   0.014
                                     bups       0.092   -0.050
                                  procexp       0.092   0
                             stderred-git       0.092   -0.020
                       ruby-kaitai-struct       0.092   0.011
                    openfx-neatvideo-demo       0.092   -0.018
                                 libupnpp       0.093   0.011
                                    advcp       0.098   0.005
                             mkchromecast       0.098   0.020
             python-qscintilla-qt4-common       0.098   0.030
                             netatop-dkms       0.099   0.025
                          python-travispy       0.100   0.027
                         poissonrecon-git       0.100   -0.001
                                   lib3ds       0.101   -0.019
                               llvm6-libs       0.101   -0.014
                      python-pychromecast       0.101   0.029
                            cmvs-pmvs-git       0.103   0.027
                       perl-extutils-xspp       0.103   -0.021
                                  gpx-git       0.104   -0.004
                                  syncany       0.104   0.022
                               fonts-tlwg       0.105   -0.011
                                  bracket       0.108   -0.037
                                 fwupdate       0.109   -0.029
                                g3dviewer       0.111   -0.008
                            python2-atspi       0.111   0.024
                         spectrum3d-debug       0.111   0.015
                               attica-qt4       0.112   0.002
                       mitsuba-blender-hg       0.114   -0.001
                   blender-plugin-bligify       0.115   -0.020
                       radare2-cutter-git       0.118   0.012
                     dolphin-megasync-git       0.118   -0.020
              perl-extutils-makemaker-aur       0.119   0.022
                                    bless       0.121   0
                                   admesh       0.121   0.011
                            liblastfm-qt4       0.121   -0.004
                     ruby-connection_pool       0.121   0.044
                                   idnkit       0.126   0.017
                                  trickle       0.126   -0.003
                                ptpython2       0.128   -0.003
                           gromit-mpx-git       0.128   0.020
                                   aurman       0.129   0.019
                                 ptpython       0.131   -0.011
                                   pacaur       0.134   0.016
                 blender-plugin-miratools       0.134   0.006
                          libvirt-python2       0.137   -0.011
                       ruby-method_source       0.138   0.056
                               polkit-qt4       0.139   -0.005
                                 libgnome       0.140   -0.013
                               ddrutility       0.141   0.011
                             aurutils-git       0.141   -0.010
                             python-pyqt4       0.141   0.021
                             ceres-solver       0.142   0.027
                           python2-pyscss       0.142   -0.040
                                monitorix       0.144   0.012
                               libglademm       0.145   0.003
                                lightzone       0.149   0
                                 megasync       0.150   0.003
                                neatimage       0.151   0.009
                                       f3       0.154   0.006
                                 amide-hg       0.154   0.004
                         python2-llvmlite       0.154   -0.044
                            repetier-host       0.156   -0.004
                                  tragtor       0.157   0.011
                              hotspot-git       0.159   -0.002
                 ruby-net-http-persistent       0.159   0.048
                       ruby-pusher-client       0.160   0.039
                                  ardesia       0.161   -0.013
                          gtk-xfce-engine       0.161   -0.014
                         python2-capstone       0.161   0
                  libdesktop-agnostic-git       0.163   0.009
                                 keepassx       0.164   -0.038
                              uswsusp-git       0.165   -0.010
                           qscintilla-qt4       0.167   0.023
                                     qwt5       0.171   0.019
                                   upplay       0.171   -0.025
                     ncurses5-compat-libs       0.171   0.003
                             devtools-git       0.174   0.030
                          libdbusmenu-qt4       0.176   0.020
                          qt-virt-manager       0.178   0.007
                          python-llvmlite       0.180   0.016
                          blitz-cppqed-hg       0.182   -0.010
                      pulseaudio-dlna-git       0.185   0.018
            blender-plugin-retopoflow-git       0.190   0.003
                                   eigen2       0.195   0.032
                                    flann       0.196   0.009
                                   gigolo       0.205   -0.004
                                  xmedcon       0.206   0.055
                      perl-math-planepath       0.206   0.002
                                     lcms       0.210   -0.006
                               partio-git       0.212   -0.009
                              autofdo-git       0.216   0.012
                                   bashdb       0.217   0.025
                                 qtwebkit       0.220   0.025
                               seexpr-qt5       0.220   0.013
                         sublime-text-dev       0.225   0.059
                      gimp-plugin-mathmap       0.225   -0.047
                                   yaourt       0.226   0.009
                 yafaray-blender-exporter       0.226   0.059
                   gnome-activity-journal       0.230   -0.005
                                  libgksu       0.237   0.062
                                 haguichi       0.238   -0.014
                                 turbovnc       0.240   -0.013
                                      djv       0.243   0.004
                              ananicy-git       0.243   0
                                     acct       0.250   0.032
                                    veles       0.252   -0.070
                              firestarter       0.256   0.055
                              oxygen-kde4       0.256   0.011
                 kaitai-struct-visualizer       0.256   0
                            python2-pecan       0.260   -0.071
                         python2-autobahn       0.262   -0.097
                                   pencil       0.267   -0.029
                          gnome-mime-data       0.273   -0.003
                               namcap-git       0.276   -0.010
                            coin-or-lemon       0.276   0.016
                               phonon-qt4       0.280   0.016
                                  ruby-gh       0.281   0.081
                      gravit-designer-bin       0.281   -0.024
                                     criu       0.283   -0.004
                                 blktrace       0.286   0.015
                          numix-gtk-theme       0.286   -0.028
                          masterpdfeditor       0.287   0.014
        gnome-shell-theme-elegance-colors       0.290   0.054
                                megatools       0.293   0.011
                          libqtelegram-ae       0.296   0.004
                               pylote-git       0.307   0.053
                              amide-debug       0.309   0.004
                                   orbit2       0.312   0.017
                                  gputest       0.312   -0.002
                                 alleyoop       0.316   0.032
                          slowmovideo-git       0.319   0.045
                        ruby-highline-1.6       0.322   0.063
                                  glc-lib       0.325   -0.069
                               lifeograph       0.325   -0.020
                             ruby-launchy       0.330   0.053
                        google-chrome-dev       0.330   -0.035
                               teamviewer       0.332   0.007
                            google-chrome       0.332   0.006
                            libgnome-data       0.345   0.003
                        peazip-gtk2-build       0.354   0.021
                                 usermode       0.362   -0.096
                   python2-prompt_toolkit       0.366   -0.148
                             luxrender-hg       0.375   -0.026
             blender-plugin-luxcorerender       0.383   0.088
                             uniconvertor       0.387   -0.007
                                    kazam       0.388   0.017
                           torrent-search       0.392   0.007
                          python2-dulwich       0.398   0.050
           blender-plugin-blenderseed-git       0.401   0.060
                                gcc7-libs       0.403   0.035
                                keepassx2       0.407   -0.062
                                    snapd       0.409   0.043
                              openmvs-git       0.410   -0.024
                      ipython-ipyparallel       0.411   0.076
                     radare2-bindings-git       0.414   0.007
                                  mve-git       0.415   -0.045
                               ruby-ethon       0.417   0.098
                                   q4wine       0.421   -0.021
                           libgnomecanvas       0.430   0.007
                                   liblas       0.432   0.019
                           ruby-websocket       0.436   0.124
                  dump1090-mutability-git       0.444   -0.019
                          boost-63-compat       0.446   -0.001
                                 dockbarx       0.458   0.048
                        ruby-typhoeus-0.6       0.476   0.177
                                      dar       0.479   0.041
                             ruby-coderay       0.503   0.096
                    teigha-file-converter       0.525   -0.010
                                franz-bin       0.546   -0.022
                               webkitgtk2       0.559   -0.075
                                  sk1libs       0.594   0.169
                                 vuurmuur       0.602   -0.016
                               cppqed-git       0.602   0.106
                              libbonoboui       0.606   0
                                   libg3d       0.608   -0.046
                                webkitgtk       0.613   0.160
                                gcc6-libs       0.618   0.045
                                  vivaldi       0.625   -0.363
                                ffmpeg2.8       0.625   -0.009
                             pyqt4-common       0.638   0.041
                                ruby-json       0.641   0.125
                                    byobu       0.643   0.041
             xcursor-chameleon-anthracite       0.648   -0.091
                                 plymouth       0.655   0.018
                              xfdashboard       0.659   0.082
                                pamac-aur       0.664   0.015
                              awesomebump       0.673   0.046
            xcursor-chameleon-darkskyblue       0.703   0.003
                              vim-ale-git       0.705   -0.049
                xcursor-chameleon-skyblue       0.726   0.104
                                 babl-git       0.730   0.015
                                libbonobo       0.761   -0.025
                             cinelerra-cv       0.765   -0.015
                  xcursor-chameleon-pearl       0.767   0.111
                               libgnomeui       0.813   0.031
                        luxcorerender-git       0.822   -0.092
       blender-plugin-animation-nodes-git       0.822   0.161
                   avant-window-navigator       0.825   -0.038
                  simplenote-electron-bin       0.858   0.096
                  xcursor-chameleon-white       0.876   0.205
                            mattercontrol       0.926   -0.157
                                     lwks       1.016   0.022
                            gstreamer0.10       1.022   0.013
                                 pfstools       1.069   -0.016
                                 smartgit       1.240   0.225
                    chromium-snapshot-bin       1.253   0.105
           blender-plugin-animation-nodes       1.253   -0.016
                              ruby-travis       1.385   0.403
                                    icesl       1.395   -0.087
                             cloudcompare       1.448   -0.005
                           ruby-backports       1.539   -0.348
                                 xnviewmp       1.585   0.100
                              radare2-git       1.946   0.111
                            eclipse-pydev       1.980   0.055
                                 gegl-git       2.014   0.072
                                 ruby-pry       2.016   0.728
                          depot-tools-git       2.073   -0.022
                                  yafaray       2.117   0.124
                                blaze-git       2.232   0.106
                           youtube-dl-git       2.289   -0.072
                                flens-git       2.342   -0.015
                            python2-scons       2.342   0.069
                                    tiger       2.355   -0.070
         blender-plugin-manuelbastionilab       2.397   -0.102
                          hdf5_18-openmpi       2.409   0.795
                                      vxl       2.591   0.180
                                     gcc6       2.618   0.021
                     python2-scikit-learn       2.657   -0.061
                              openmvg-git       2.665   -0.028
                           python2-pandas       2.672   -0.117
                              osp-tracker       2.766   0.137
                             inkscape-git       2.793   -0.082
                                    gamin       2.795   0.157
                                openboard       2.832   0.120
                              blender-2.7       2.858   -0.052
                             obs-v4l2sink       2.915   -0.118
                                  volpack       3.062   -0.214
                                 bcompare       3.078   0.019
                          blender-2.8-git       3.086   0.114
                                     gcc7       3.556   0.084
                          indigo-renderer       3.675   0.805
                               visual-sfm       3.680   -0.101
                                    dcmtk       4.139   0.172
                                      u3d       4.338   0.416
                                    gcc49       4.489   0.005
                              meshlab-git       4.526   -0.223
                      collada-dom-mitsuba       4.592   -0.022
                                    gcc54       4.789   0.158
                              mitsuba-git       4.884   -0.239
                            python-plotly       5.638   0.735
                                 gcc6-gcj       5.695   0.233
                            python2-pyqwt       5.984   -15.233
                            systemtap-git       6.060   0.415
                                      qt4       6.086   1.297
              numix-circle-icon-theme-git       6.187   -0.313
                                  kdelibs       6.894   -0.125
                   visual-studio-code-bin       7.119   0.555
                                 linux-pf       7.288   -0.067
                                     vtk6       7.674   0.057
                                kitematic       8.784   -0.396
                                      f2c       8.835   0.594
                                gnome-vfs       8.871   0.080
                                 peerflix       8.929   -0.031
                         gimp-develop-git       9.118   0.229
                                 cutegram       10.187  -0.425
                          atom-editor-bin       11.328  0.119
                             python2-wnck       11.866  0.051
                       python2-keybinder2       11.951  0.291
                         teighaviewer-qt5       12.734  0.288
                                    veusz       12.738  -6.640
                         perl-wx-glcanvas       12.802  -0.526
                                      pcl       13.020  0.248
                       faience-icon-theme       13.074  0.121
                             python-numba       13.355  -49.896
       perl-math-convexhull-monotonechain       13.417  0.398
                      perl-linux-inotify2       13.809  0.550
                                 codelite       13.825  0.325
                            python2-numba       14.037  -47.605
              perl-javascript-minifier-xs       14.205  -0.268
                                      oce       15.960  0.016
                             opencascade7       16.367  0.338
                           perl-math-libm       16.845  0.341
                           popcorntime-ce       17.467  0.220
                         pygtksourceview2       18.228  -0.253
                                  libuser       18.350  -0.250
                               pyalpm-git       18.775  -0.220
                                  dropbox       19.250  3.741
                             mint-x-icons       20.451  -0.472
                               netgen-git       21.767  0.956
               perl-math-geometry-voronoi       23.298  0.252
                            bcompare-mate       24.805  -0.720
                             google-earth       24.848  -1.718
                              perl-opengl       25.059  -0.330
                               slic3r-git       25.486  -0.355
                        bcompare-nautilus       25.582  -0.452
                        bcompare-cinnamon       25.858  0.831
                                 brackets       28.300  -0.064
                        perl-math-clipper       28.515  -7.875
                          bcompare-thunar       28.860  -0.416
                                  rsyslog       29.207  4.091
                     numix-icon-theme-git       31.547  0.356
                                     lash       31.658  3.451
                         perl-threads-aur       33.143  1.499
                         linux-pf-headers       33.319  -0.697
                         fusecompress-git       34.907  0.589
                               opengv-git       36.048  -0.150
                            appleseed-git       46.444  -0.929
                         python-sip-pyqt4       50.444  -6.191
                        python2-sip-pyqt4       52.306  -5.552
                                  perl-wx       55.486  -182.958
                               perl-event       63.703  0.276
                       haskell-semigroups       123.789 3.112
                             paraview-opt       165.556 -3.572
comparisons:                                    A       A-B

                                   [sec]:       1728.401        -303.100
@bartoszek

This comment has been minimized.

Copy link
Contributor Author

bartoszek commented Dec 5, 2019

What do you mean by this, a run of checkrebuild will create this file, and some other hook (pacman hook?) will just look at the presence of that file?

Currently rebuild-detector.hook check all packages (foreign+local_repo) every time it's called. This is a bit wasteful behavior as if package is once detected as needs rebuild there's no need to test it again untill it's updated. So basically each rune of rebuild-detect.hook we make a file containing packages flagged as need rebuild and omit them during next check. After package gets update ( passed as stdin from NeedsTargets ) we recheck it and remove from flagged packages if checks up.

@maximbaz

This comment has been minimized.

Copy link
Owner

maximbaz commented Feb 3, 2020

Hey, sorry for the long response from my side.

I've looked again into this, and while I can see the desire to get the tool produce perfect results, the sheer amount of code and complexity makes me uncomfortable to merge this - it will be hard for me to maintain (and frankly even understanding how this works exactly is quite difficult). I also didn't particularly like making this tool stateful, but I do like some other ideas that I will be taking into the codebase.

Thanks again for creating the PR and answering my questions, I hope you don't mind using your own fork if you still use this paranoid mode 🙂

@maximbaz maximbaz closed this Feb 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

2 participants
You can’t perform that action at this time.