NonDurable cursor for managed ledger #366
Conversation
merlimat
added
the
type/enhancement
merlimat
force-pushed
the
non-durable-cursor
branch
from
1eb6639
to
fbd99f4
Compare
merlimat
force-pushed
the
non-durable-cursor
branch
from
fbd99f4
to
98e687e
Compare
| @@ -135,6 +135,23 @@ | |||
| public ManagedCursor openCursor(String name) throws InterruptedException, ManagedLedgerException; | |||
|
|
|||
| /** | |||
| * Creates a new cursor whose metadata is not backed by durable storage. A caller can treat the non-durable cursor | |||
| * exactly like a normal cursor, with the only difference in that after restart it will not remember which entries | |||
| * were deleted. Also it does not prevent data from being deleted. | |||
There was a problem hiding this comment.
Also it does not prevent data from being deleted.
Isn't it create unexpected behavior for client when slowest durable cursor moves to next ledger and ml trims the ledger. Let's say if this cursor is slower than other durable cursor then ManagedLedger will immediately delete data once durable-cursor consumeAndAck from that ml-ledger and switch the ledger, so non-durable cursor may not get expected chunk of data in stream.
- Instead can't we treat non-durable cursor like durable and not delete until it consume messages. Because if non-durable cursor/consumer disconnects then broker is anyway removes this non-durable cursor and can move position according to durable-cursor's
markDeletePosition.
| public class PositionImpl implements Position, Comparable<PositionImpl> { | ||
|
|
||
| private final long ledgerId; | ||
| private final long entryId; | ||
|
|
||
| public static Position earliest = new PositionImpl(-1, -1); | ||
| public static Position latest = new PositionImpl(Long.MAX_VALUE, Long.MAX_VALUE); |
There was a problem hiding this comment.
MIN_POSITION / MAX_POSITION?
| checkManagedLedgerIsOpen(); | ||
| checkFenced(); | ||
|
|
||
| return new NonDurableCursorImpl(bookKeeper, config, this, null, (PositionImpl) startCursorPosition); |
There was a problem hiding this comment.
for debugging, should broker generate or expect cursor-name or id rather passing null?
|
@rdhabalia Maybe the name About non impeding messages to be deleted, consider that in case of non-durable topic, during a disconnection the cursor will go away and data will get potentially deleted anyway. So I prefer it to be explicitly ingrained into the API. Same thing about naming the cursor. It will go away in any case after a restart, so I don't see the advantage of naming it. When data gets deleted, the cursor will just skip over it. The intended usage for the non-durable cursor and topic reader is in conjunction with the message retention, to make sure data sticks around for the intended amount of time |
Fixes apache#366 ### Motivation In current code of `pulsar/internal/connection.go` we have 2 channels, closeCh and incomingRequestsCh. when the connection closes, the current mis-use of these 2 channels may have a deadlock. PR apache#366 has detailed steps to reproduce and the root cause [analysis](apache/pulsar-client-go#366 (comment)) . This PR tries to fix the deadlock. ### Modifications - make the close logic independent, not in the same loop of normal events handling. - when the connection closed, handle the existing requests in the channel and return an error to avoid deadlock. ### Verifying this change passed the tests in apache#366 current ut passed
Motivation
As part of #355, ManagedLedger needs to support a lightweight cursor that can be just used for reading from a specific initial position.
Modifications
Introduce new ML API to create a non-durable cursor. This cursor is exactly the same as a regular cursor except in that it doesn't persist the mark-delete position (or the individually deleted messages). Also, it doesn't prevent entries to be deleted.
Result
Can implement topic reader by plugging a non-durable cursor into a regular broker subscription and dispatcher objects.