Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Parallel fetch #1704

Closed
wants to merge 3 commits into from
Closed

WIP: Parallel fetch #1704

wants to merge 3 commits into from

Conversation

Zevenberge
Copy link
Contributor

Running CI tests to debug segfaults

@dlang-bot
Copy link
Collaborator

dlang-bot commented May 11, 2019 •
edited
Loading

Thanks for your pull request and interest in making D better, @Zevenberge! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

  • My PR is fully covered with tests (you can see the coverage diff by visiting the details link of the codecov check)
  • My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
  • I have provided a detailed rationale explaining my changes
  • New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

@Zevenberge
Copy link
Contributor Author 

$ ./issue1524-maven-upgrade-dependency-tree.sh
+ . ./common.sh
++ SOURCE_FILE=-x
++ set -ueEo pipefail
++ trap 'die $LINENO' ERR
+ PORT=23366
+ dub remove maven-dubpackage-a --non-interactive '--version=*'
+ true
+ dub remove maven-dubpackage-b --non-interactive '--version=*'
+ true
+ gdb -batch -ex run -ex bt -args /home/marco/d/dub/bin/dub build --single ./test_registry.d
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff1e6d700 (LWP 22354)]
[Thread 0x7ffff1e6d700 (LWP 22354) exited]
[New Thread 0x7ffff1e6d700 (LWP 22355)]
[New Thread 0x7ffff0950700 (LWP 22356)]
[New Thread 0x7fffe3fff700 (LWP 22357)]
[New Thread 0x7fffea8ff700 (LWP 22358)]
[New Thread 0x7fffea0fe700 (LWP 22359)]
[New Thread 0x7fffe98fd700 (LWP 22360)]
Fetching botan-math 1.0.3 (getting selected version)...
Fetching memutils 0.4.13 (getting selected version)...
Fetching vibe-core 1.6.2 (getting selected version)...
Fetching taggedalgebraic 0.11.4 (getting selected version)...
Fetching stdx-allocator 2.77.5 (getting selected version)...
Fetching vibe-d 0.8.5 (getting selected version)...
Fetching botan 1.12.10 (getting selected version)...
[New Thread 0x7fffe90fc700 (LWP 22361)]
[New Thread 0x7fffe37fe700 (LWP 22363)]
[New Thread 0x7fffe88fb700 (LWP 22362)]
[New Thread 0x7fffe2ffd700 (LWP 22364)]
[New Thread 0x7fffe1ffb700 (LWP 22365)]
[New Thread 0x7fffe27fc700 (LWP 22366)]
[New Thread 0x7fffe17fa700 (LWP 22367)]
[Thread 0x7fffe27fc700 (LWP 22366) exited]
[Thread 0x7fffe88fb700 (LWP 22362) exited]
[Thread 0x7fffe90fc700 (LWP 22361) exited]
[Thread 0x7fffe1ffb700 (LWP 22365) exited]
[Thread 0x7fffe37fe700 (LWP 22363) exited]
[Thread 0x7fffe2ffd700 (LWP 22364) exited]
[Thread 0x7fffe17fa700 (LWP 22367) exited]
[New Thread 0x7fffe17fa700 (LWP 22370)]
[New Thread 0x7fffe27fc700 (LWP 22371)]
[Thread 0x7fffe17fa700 (LWP 22370) exited]
[Thread 0x7fffe27fc700 (LWP 22371) exited]
[New Thread 0x7fffe1ffb700 (LWP 22372)]
[New Thread 0x7fffe37fe700 (LWP 22373)]
[Thread 0x7fffe37fe700 (LWP 22373) exited]
[Thread 0x7fffe1ffb700 (LWP 22372) exited]
[New Thread 0x7fffe37fe700 (LWP 22374)]
[New Thread 0x7fffe1ffb700 (LWP 22375)]
[Thread 0x7fffe1ffb700 (LWP 22375) exited]
[Thread 0x7fffe37fe700 (LWP 22374) exited]
[New Thread 0x7fffe1ffb700 (LWP 22376)]
[Thread 0x7fffe1ffb700 (LWP 22376) exited]
[New Thread 0x7fffe1ffb700 (LWP 22377)]
[New Thread 0x7fffe37fe700 (LWP 22378)]
[New Thread 0x7fffe27fc700 (LWP 22379)]
[Thread 0x7fffe37fe700 (LWP 22378) exited]
[Thread 0x7fffe1ffb700 (LWP 22377) exited]
[Thread 0x7fffe27fc700 (LWP 22379) exited]
[New Thread 0x7fffe27fc700 (LWP 22380)]
[Thread 0x7fffe27fc700 (LWP 22380) exited]
[New Thread 0x7fffe27fc700 (LWP 22381)]
[Thread 0x7fffe27fc700 (LWP 22381) exited]
[New Thread 0x7fffe37fe700 (LWP 22382)]
[New Thread 0x7fffe27fc700 (LWP 22383)]
[Thread 0x7fffe37fe700 (LWP 22382) exited]
[Thread 0x7fffe27fc700 (LWP 22383) exited]
Fetching diet-ng 1.5.0 (getting selected version)...
[New Thread 0x7fffe27fc700 (LWP 22387)]
Fetching openssl 1.1.6+1.0.1g (getting selected version)...
[New Thread 0x7fffe37fe700 (LWP 22388)]
[Thread 0x7fffe27fc700 (LWP 22387) exited]
[Thread 0x7fffe37fe700 (LWP 22388) exited]
Fetching eventcore 0.8.42 (getting selected version)...
[New Thread 0x7fffe37fe700 (LWP 22389)]
[Thread 0x7fffe37fe700 (LWP 22389) exited]
[New Thread 0x7fffe37fe700 (LWP 22390)]
[Thread 0x7fffe37fe700 (LWP 22390) exited]
[New Thread 0x7fffe37fe700 (LWP 22391)]
[New Thread 0x7fffe27fc700 (LWP 22392)]
Fetching mir-linux-kernel 1.0.1 (getting selected version)...
[New Thread 0x7fffe1ffb700 (LWP 22393)]
[Thread 0x7fffe37fe700 (LWP 22391) exited]
[Thread 0x7fffe27fc700 (LWP 22392) exited]
[Thread 0x7fffe1ffb700 (LWP 22393) exited]
Fetching libevent 2.0.2+2.0.16 (getting selected version)...
[New Thread 0x7fffe27fc700 (LWP 22394)]
[Thread 0x7fffe27fc700 (LWP 22394) exited]
[New Thread 0x7fffe27fc700 (LWP 22395)]
[New Thread 0x7fffe37fe700 (LWP 22396)]
[New Thread 0x7fffe1ffb700 (LWP 22397)]
[Thread 0x7fffe1ffb700 (LWP 22397) exited]
[Thread 0x7fffe37fe700 (LWP 22396) exited]
[Thread 0x7fffe27fc700 (LWP 22395) exited]
[New Thread 0x7fffe17fa700 (LWP 22398)]
[Thread 0x7fffe17fa700 (LWP 22398) exited]
[New Thread 0x7fffe90fc700 (LWP 22399)]
[Thread 0x7fffe90fc700 (LWP 22399) exited]
[New Thread 0x7fffe1ffb700 (LWP 22400)]
[New Thread 0x7fffe37fe700 (LWP 22401)]
[Thread 0x7fffe37fe700 (LWP 22401) exited]
[Thread 0x7fffe1ffb700 (LWP 22400) exited]
Fetching libasync 0.8.4 (getting selected version)...
[New Thread 0x7fffe37fe700 (LWP 22403)]
[Thread 0x7fffe37fe700 (LWP 22403) exited]
[New Thread 0x7fffe37fe700 (LWP 22404)]
[Thread 0x7fffe37fe700 (LWP 22404) exited]
[New Thread 0x7fffe37fe700 (LWP 22411)]
[Thread 0x7fffe37fe700 (LWP 22411) exited]

Thread 5 "dub" received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7fffe3fff700 (LWP 22357)]
pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185	../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00000000009d5c7d in core.sync.condition.Condition.wait() ()
#2  0x0000000000993597 in std.parallelism.TaskPool.wait() ()
#3  0x0000000000993105 in _D3std11parallelism8TaskPool3popMFZPSQBjQBi12AbstractTask ()
#4  0x0000000000993087 in std.parallelism.TaskPool.executeWorkLoop() ()
#5  0x0000000000993061 in std.parallelism.TaskPool.startWorkLoop() ()
#6  0x0000000000978e20 in core.thread.Thread.run() ()
#7  0x00000000009d542c in thread_entryPoint ()
#8  0x00007ffff7bc16ba in start_thread (arg=0x7fffe3fff700) at pthread_create.c:333
#9  0x00007ffff72d541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
+ PID=22417
+ sleep 1
+ ./test_registry --folder=./issue1524-maven-upgrade-dependency-tree --port=23366
[main(----) INF] Listening for requests on http://[::]:23366/
Failed to listen on 0.0.0.0:23366
+ trap 'kill $PID 2>/dev/null || true' exit
+ echo 'Trying to download maven-dubpackage-a (1.0.5) with dependency to maven-dubpackage-b (1.0.6)'
Trying to download maven-dubpackage-a (1.0.5) with dependency to maven-dubpackage-b (1.0.6)
+ /home/marco/d/dub/bin/dub upgrade --root ./issue1524-maven-upgrade-dependency-tree --skip-registry=standard --registry=mvn+http://localhost:23366/maven/release/dubpackages
Upgrading project in /home/marco/d/dub/test/issue1524-maven-upgrade-dependency-tree
Fetching maven-dubpackage-a 1.0.5 (getting selected version)...
Fetching maven-dubpackage-b 1.0.6 (getting selected version)...
+ dub remove maven-dubpackage-a --non-interactive --version=1.0.5
Removing maven-dubpackage-a in /home/marco/.dub/packages/maven-dubpackage-a-1.0.5/maven-dubpackage-a/
Removed package: 'maven-dubpackage-a'
Removed maven-dubpackage-a, version 1.0.5.
+ dub remove maven-dubpackage-b --non-interactive --version=1.0.6
Removing maven-dubpackage-b in /home/marco/.dub/packages/maven-dubpackage-b-1.0.6/maven-dubpackage-b/
Removed package: 'maven-dubpackage-b'
Removed maven-dubpackage-b, version 1.0.6.
+ kill 22417
[main(----) INF] Received signal 15. Shutting down.
Warning (thread: main): leaking eventcore driver because there are still active handles
   FD 7 (streamListen)
Warning (thread: main): leaking eventcore driver because themarco@lubuntu26:~/d/dub/test$ re are still active handles
   FD 7 (streamListen)

@wilzbach
Copy link
Member

Looks like we have to roll our own simpler worker, see e.g. https://tour.dlang.org/tour/en/multithreading/message-passing

wilzbach
wilzbach reviewed May 28, 2019
View reviewed changes
source/dub/dub.d
import std.parallelism : TaskPool, totalCPUs;
// Avoid being blocked by a slow download (even on a single-threaded machines)
auto taskPool = new TaskPool(totalCPUs.clamp(3, 6));
foreach (p; taskPool.parallel(versions.byKey, 1)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have we tried to use .byKey array here?

Copy link
Contributor Author

@Zevenberge Zevenberge May 28, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, not yet. We will want to do that in the final version.

During the experimenting, I reset the branch to start from your implementation. I noticed that, depending on the amount of logging I added, the behaviour was changing. Of course, as a race condition occurs, this is to be expected. I didn´t expect it would corrupt the array of dependencies (it once failed with an empty list of dependencies). The sanity check after the fetch would fail (see the source/dub/project.d:159). It might be the same cause (as in, the same read/write operation), but it might not be.

@Zevenberge
Copy link
Contributor Author

@wilzbach I think I've figured it out. There were a few data races in the package manager. Currently synchronized's are shattered throughout the package manager class. What would be the idiomatic way of solving this? Making m_packageManager in Dub shared?

Synchronized fetch
b9b2a10 
Added some careful logging.


Loading the packages is now synchronized


Removed lock token from synchronized

Shared and synchronized


Revert "Added some careful logging."

This reverts commit 858ebb8.

Updating m_project is now synchronized


Removed synchronized from fetch

Synchronized in remove pakcage manager


Using this instead of a lock object
@Geod24
Copy link
Member

Geod24 commented Jun 18, 2024

Closing as stalled. The fetch logic has changed recently. Parallel fetch would still be good, but would probably mean restarting this from scratch.

@Geod24 Geod24 closed this Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wilzbach wilzbach wilzbach left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Zevenberge @dlang-bot @wilzbach @Geod24