-
-
Notifications
You must be signed in to change notification settings - Fork 244
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
1202: prioritized merge #1205
1202: prioritized merge #1205
Changes from 1 commit
ea61f60
0b1c831
c31beda
0e5844a
1b6c1c4
00f57ee
88e6a04
9b04ba4
affe39e
ec435e1
99cc03d
df122f7
c23d1cf
4d52bfc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,157 @@ | ||
/* | ||
* Copyright (c) 2014-2020 by The Monix Project Developers. | ||
* See the project homepage at: https://monix.io | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package monix.reactive.internal.builders | ||
|
||
import java.util.concurrent.PriorityBlockingQueue | ||
|
||
import monix.execution.Ack.{Continue, Stop} | ||
import monix.execution.cancelables.CompositeCancelable | ||
import monix.execution.{Ack, Cancelable, Scheduler} | ||
import monix.reactive.Observable | ||
import monix.reactive.observers.Subscriber | ||
|
||
import scala.concurrent.{Future, Promise} | ||
import scala.jdk.CollectionConverters._ | ||
import scala.util.Success | ||
|
||
private[reactive] final class MergePrioritizedListObservable[A](sources: Seq[Observable[A]], priorities: Seq[Int]) | ||
extends Observable[A] { | ||
require(sources.size == priorities.size, "sources.size != priorities.size") | ||
|
||
override def unsafeSubscribeFn(out: Subscriber[A]): Cancelable = { | ||
import out.scheduler | ||
|
||
val numberOfObservables = sources.size | ||
|
||
val lock = new AnyRef | ||
var isDone = false | ||
|
||
// NOTE: We use arrays and other mutable structures here to be as performant as possible. | ||
|
||
// MUST BE synchronized by `lock` | ||
var lastAck = Continue: Future[Ack] | ||
|
||
case class PQElem(data: A, promise: Promise[Ack], priority: Int) extends Comparable[PQElem] { | ||
override def compareTo(o: PQElem): Int = | ||
priority.compareTo(o.priority) | ||
} | ||
|
||
val pq = new PriorityBlockingQueue[PQElem](sources.size) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we use normal There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure, I will look at replacing it and adding necessary synchronization, then adding tests. Otherwise it looks good? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry, I haven't got a moment to take a close look, but it seems to look good. I think it's not too far from
If you don't feel like doing it - that's okay, It might also be nice to add a benchmark vs normal merge, but that's just my curiosity and completely optional. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since updating |
||
|
||
// MUST BE synchronized by `lock` | ||
var completedCount = 0 | ||
|
||
// MUST BE synchronized by `lock` | ||
def rawOnNext(a: A): Future[Ack] = { | ||
if (isDone) Stop else out.onNext(a) | ||
} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added the above logic to ensure that we complete the upstream onNext promises when the downstream stops or errors early while we still have pending upstream items. Added tests to verify as well. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could use our There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Much nicer, thanks. |
||
|
||
def processNext(): Future[Ack] = { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. needs a comment: There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed |
||
val e = pq.remove() | ||
val fut = rawOnNext(e.data) | ||
e.promise.completeWith(fut) | ||
fut | ||
} | ||
|
||
// MUST BE synchronized by `lock` | ||
def signalOnNext(): Future[Ack] = | ||
lock.synchronized { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. redundant There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed |
||
lastAck = lastAck match { | ||
case Continue => processNext() | ||
case Stop => Stop | ||
case async => | ||
async.flatMap { | ||
// async execution, we have to re-sync | ||
case Continue => lock.synchronized(processNext()) | ||
case Stop => Stop | ||
} | ||
} | ||
|
||
lastAck | ||
} | ||
|
||
def signalOnError(ex: Throwable): Unit = | ||
lock.synchronized { | ||
if (!isDone) { | ||
isDone = true | ||
out.onError(ex) | ||
lastAck = Stop | ||
completePromises() | ||
} | ||
} | ||
|
||
def signalOnComplete(): Unit = | ||
lock.synchronized { | ||
completedCount += 1 | ||
|
||
if (completedCount == numberOfObservables && !isDone) { | ||
lastAck match { | ||
case Continue => | ||
isDone = true | ||
out.onComplete() | ||
completePromises() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Regarding the issue:
Looking at the contract: https://monix.io/docs/3x/reactive/observers.html#contract
It implies that Source B could call: out.onNext(lastElem)
out.onComplete() In that case, we could send remaining elements from the queue to the downstream instead of completing each promise with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
So long as the downstream returns
|
||
case Stop => | ||
() // do nothing | ||
case async => | ||
async.onComplete { | ||
case Success(Continue) => | ||
lock.synchronized { | ||
if (!isDone) { | ||
isDone = true | ||
out.onComplete() | ||
completePromises() | ||
} | ||
} | ||
case _ => | ||
() // do nothing | ||
} | ||
} | ||
|
||
lastAck = Stop | ||
} | ||
} | ||
|
||
def completePromises(): Unit = { | ||
pq.iterator().asScala.foreach(e => e.promise.complete(Success(Stop))) | ||
} | ||
|
||
val composite = CompositeCancelable() | ||
|
||
sources.zip(priorities).foreach { pair => | ||
composite += pair._1.unsafeSubscribeFn(new Subscriber[A] { | ||
implicit val scheduler: Scheduler = out.scheduler | ||
|
||
def onNext(elem: A): Future[Ack] = { | ||
if (isDone) { | ||
Stop | ||
} else { | ||
pq.add(PQElem(elem, Promise(), pair._2)) | ||
signalOnNext() | ||
} | ||
} | ||
|
||
def onError(ex: Throwable): Unit = | ||
signalOnError(ex) | ||
|
||
def onComplete(): Unit = | ||
signalOnComplete() | ||
}) | ||
} | ||
composite | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very interesting implementation @ctoomey.
I do have one suggestion about this signature ... it's not clear that the
priorities
should match the sources and there's no reason for it.How about ...
Then you could do ...
That way the signature is clearer, and you don't need that runtime
assert
in the implementation.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ya I'd prefer that signature too and in fact had it that way originally, but decided to match the way Akka streams did it. I'll change it.