FEAT: reserve/publish work-flow API #415

Closed
pbalcer opened this Issue Jan 27, 2017 · 4 comments

Comments

Projects
None yet
3 participants
@pbalcer

pbalcer commented Jan 27, 2017

Reserve/Initialize/Publish work-flow

Rationale

The research community proposes a different, more flexible programming models. It's objectively better in some ways and allows for new programming constructs. All of the elements are mostly in place and the new model can be introduced fairly quickly.

Description

Implement new functionality enabled by #380 feature.
A new pmemobj_memops opaque structure will be introduced that contains reservation. A single memops structure can contain many reservations (up to a limit) that can be published either in an atomic way or inside of a transaction.
In the future the pmemobj_memops interface may be extended to allow for deferred freeing and plain memory modifications. This would allow the user to create a very rudimentary redo log transactions.

API Changes

typedef struct pmemobj_memops;

PMEMoid pmemobj_reserve(PMEMobjpool *pop, struct pmemobj_memops *ops, PMEMoid *oidp, size_t size, uint64_t type_num);
int pmemobj_publish(PMEMobjpool *pop, struct pmemobj_memops *ops);
int pmemobj_tx_publish(struct pmemobj_memops *ops);
int pmemobj_cancel(PMEMobjpool *pop, PMEMoid reservation);

Implementation details

TBD

@pbalcer pbalcer added this to the 1.4 milestone Jan 27, 2017

@pbalcer pbalcer self-assigned this Jan 27, 2017

@marcinslusarz

This comment has been minimized.

Show comment
Hide comment
@marcinslusarz

marcinslusarz Jan 27, 2017

Member

I don't like the "publish" part of the API. "publish" is too vague and can be understood as making an allocation visible. Which is not what is going on.
If "reserve" allocates only in transient structures, then what "publish" does is "persisting" (name already taken), or "committing with current transaction" (name too long) or just "performing" an allocation.
Not sure.

Member

marcinslusarz commented Jan 27, 2017

I don't like the "publish" part of the API. "publish" is too vague and can be understood as making an allocation visible. Which is not what is going on.
If "reserve" allocates only in transient structures, then what "publish" does is "persisting" (name already taken), or "committing with current transaction" (name too long) or just "performing" an allocation.
Not sure.

@Maciuch

This comment has been minimized.

Show comment
Hide comment
@Maciuch

Maciuch Mar 7, 2017

Looking at the two sentences:
"A single memops structure can contain many reservations (up to a limit)"
"publishing, performed at the pre-commit phase or once the number of reserved allocations reaches either a predefined threshold or the redo log maximum capacity." (from #380)
Does it mean, that in case of having intense reservation usage that crosses threshold we'll see flushing part of the app data without it's knowledge of it?
What about a case in which you have this transparent flush, and platform dies. After restart will NVML be able to roll back and clean all the reservation data including those that crossed the threshold before? If not we can end up with a sort of silent data corruption due to a partial data presence.

Maciuch commented Mar 7, 2017

Looking at the two sentences:
"A single memops structure can contain many reservations (up to a limit)"
"publishing, performed at the pre-commit phase or once the number of reserved allocations reaches either a predefined threshold or the redo log maximum capacity." (from #380)
Does it mean, that in case of having intense reservation usage that crosses threshold we'll see flushing part of the app data without it's knowledge of it?
What about a case in which you have this transparent flush, and platform dies. After restart will NVML be able to roll back and clean all the reservation data including those that crossed the threshold before? If not we can end up with a sort of silent data corruption due to a partial data presence.

@pbalcer

This comment has been minimized.

Show comment
Hide comment
@pbalcer

pbalcer Mar 8, 2017

Not sure if I follow.
We reserve memory (in volatile state), and once we are ready to publish it, we build-up a redo log. This redo log guarantees that a single "publish" invocation is fail safe atomic. As for transactions, I did not remove the undo log, if that's what you are asking. Each TX_ALLOC 'reserves' the memory (instead of allocating it), and once we cross a threshold, we 'publish' it to the undo log. If we crash anywhere before transaction is committed, everything is simply freed from the undo logs, and the reservations simply vanish (as they were reserved from the volatile state).
This is all transparent to the application.

pbalcer commented Mar 8, 2017

Not sure if I follow.
We reserve memory (in volatile state), and once we are ready to publish it, we build-up a redo log. This redo log guarantees that a single "publish" invocation is fail safe atomic. As for transactions, I did not remove the undo log, if that's what you are asking. Each TX_ALLOC 'reserves' the memory (instead of allocating it), and once we cross a threshold, we 'publish' it to the undo log. If we crash anywhere before transaction is committed, everything is simply freed from the undo logs, and the reservations simply vanish (as they were reserved from the volatile state).
This is all transparent to the application.

@Maciuch

This comment has been minimized.

Show comment
Hide comment
@Maciuch

Maciuch Mar 8, 2017

OK, I did not catch the detail of having another log around this mechanism. Looks fine then!

Maciuch commented Mar 8, 2017

OK, I did not catch the detail of having another log around this mechanism. Looks fine then!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment