scan_partitioner is not working as expected.

[sithhell]

Parallel algorithms using the scan partitioner are spuriously failing when
the continuation in dataflow is executed asynchronously.

It seems that when passing hpx::launch::sync to the dataflow invocation works around that issue.

A fix needs to be found such that it works with (async) executors again.

[taeguk]

How is it progressed now?
Because of this issue, parallel algorithms that use scan_partitioner are very slow.
What problem does it occurs in detail to use policy.executor() instead of hpx::launch::sync?
I want to try to resolve this issue.

[hkaiser]

What problem does it occurs in detail to use policy.executor() instead of hpx::launch::sync ?

We see random segfaults in this case. Apparently things are going out of scope too early.

I want to try to resolve this issue.

Sure, go ahead - please be our guest.

[taeguk]

@hkaiser Can I get more informations about this issue?
I couldn't find the problem with looking the code.
Can you tell when the problem is happening? (ex, which parallel algorithms is executing when the segfault occurs?)

[hkaiser]

@sithhell: do you have any recollection of how to reproduce the original problem?

[taeguk]

@hkaiser One suspicion:

hpx/hpx/parallel/util/scan_partitioner.hpp

Lines 107 to 121 in ed30662

	for(auto const& elem: shape)
	{
	FwdIter it = hpx::util::get<0>(elem);
	std::size_t size = hpx::util::get<1>(elem);
	hpx::shared_future<Result1> prev = workitems.back();
	auto curr = execution::async_execute(
	policy.executor(), f1, it, size).share();
	finalitems.push_back(dataflow(hpx::launch::sync,
	f3, it, size, prev, curr));
	workitems.push_back(dataflow(hpx::launch::sync,
	f2, prev, curr));
	}

Is there no possibility about dangling reference to prev and curr?
If prev or curr is passed by reference to f2 or f3, dangling reference problem can be occurred when we use policy.executor() instead of hpx::launch::sync.

[hkaiser]

@taeguk, @heller: has this been resolved now?

[taeguk]

@hkaiser I don't know. I have no test code which reproduces original problem. So I can't try to resolve this issue. Can I get the code which reproduces the problem?

[hkaiser]

@taeguk, @sithhell Let's close this for now. We can always reopen it, if needed.

[sithhell]

Reopening again. The issue hasn't been worked ok. The partitioner still doesn't use the passed executors.

[taeguk]

@sithhell Can you give me the test cases that makes scan_partitioner strange?
I really want to address this issue. But I have no test code which reproduces original problem.

[sithhell]

sorry for not getting back to you earlier ...
If you look at https://github.com/STEllAR-GROUP/hpx/blob/master/hpx/parallel/util/scan_partitioner.hpp,
there are a few places where we use dataflow(hpx::launch::sync, ...) instead of making use of the passed executor (The PR mentioned above should point you to those places). By changing that to use the executors, you should see a failure in the tests using the scan partitioner, it's a race condition, so you will need to run a few samples.

[taeguk]

@sithhell I can't reproduce the problem.
I tested the unit tests that use scan_partitioner in hpx repository with various random seeds.
But I can't see strange things or fails of the tests.
Do you have any tests that reproduce the problem? And can you check one more whether this problem still exists in now?

[taeguk]

Reopen because of #3136

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed. Please re-open if necessary.