Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core, orderwatch, meshdb: Implement a dynamically decreasing max expiration time for orders #450

Merged
merged 40 commits into from Oct 24, 2019

Conversation

@albrow
Copy link
Member

albrow commented Oct 16, 2019

Fixes #431.

This PR is still a WIP but I'm opening it now so we can discuss the progress so far. This is an implementation of the idea we've been discussing on Slack and a solution to the problem of finite database storage. To quickly re-summarize the idea:

When storage space is plentiful, we start without any kind of limit on expiration time. As the database fills up, we enforce a limit on the maximum expiration time. Any orders that expire after that time will be removed. If more orders keep coming in we can lower the expiration time and remove more orders.

So far this PR adds an index on ExpirationTime and implements a new method called TrimOrdersByExpirationTime which contains the bulk of the core logic for this feature. There are still some challenges around how to use this function and when to call it, but the approach is looking very feasible as of now. We also still need to determine when and how quickly to increase the max expiration time after storage frees up.

@albrow albrow force-pushed the feature/dynamic-max-expiration branch from 48fcd11 to f5896bb Oct 16, 2019
constants/constants.go Outdated Show resolved Hide resolved
constants/constants.go Outdated Show resolved Hide resolved
core/core.go Outdated Show resolved Hide resolved
zeroex/order.go Outdated Show resolved Hide resolved
zeroex/orderwatch/order_watcher.go Outdated Show resolved Hide resolved
zeroex/orderwatch/order_watcher.go Show resolved Hide resolved
zeroex/orderwatch/order_watcher.go Show resolved Hide resolved
zeroex/orderwatch/order_watcher.go Outdated Show resolved Hide resolved
@fabioberger fabioberger force-pushed the development branch from 11e0fa4 to cdece2c Oct 18, 2019
@fabioberger fabioberger force-pushed the feature/dynamic-max-expiration branch from a11d5aa to 6eb6010 Oct 18, 2019
@albrow albrow force-pushed the feature/dynamic-max-expiration branch 2 times, most recently from 511dc61 to 3c32584 Oct 18, 2019
browser/ts/index.ts Outdated Show resolved Hide resolved
@albrow albrow force-pushed the feature/dynamic-max-expiration branch 2 times, most recently from 6d02665 to b7d3fa6 Oct 22, 2019
@albrow albrow changed the title WIP: Implement a dynamically decreasing max expiration time for orders Implement a dynamically decreasing max expiration time for orders Oct 22, 2019
@albrow albrow changed the title Implement a dynamically decreasing max expiration time for orders core, orderwatcher: Implement a dynamically decreasing max expiration time for orders Oct 22, 2019
@albrow albrow changed the title core, orderwatcher: Implement a dynamically decreasing max expiration time for orders core, orderwatch: Implement a dynamically decreasing max expiration time for orders Oct 22, 2019
@albrow albrow changed the title core, orderwatch: Implement a dynamically decreasing max expiration time for orders core, orderwatch, meshdb: Implement a dynamically decreasing max expiration time for orders Oct 22, 2019
@albrow

This comment has been minimized.

Copy link
Member Author

albrow commented Oct 22, 2019

@fabioberger Removed WIP tag. This PR is ready for final review.

I do eventually want to optimize our SlowCounter implementation to reduce memory allocations and math/big operations. Our current strategy of using a polling loop inside of OrderWatcher to check if the max expiration time can be increased is also inefficient, but should be good enough for now.

@albrow

This comment has been minimized.

Copy link
Member Author

albrow commented Oct 22, 2019

Just reminding myself to update the CHANGELOG after we include everything there for the upcoming 5.1.0-beta launch.

Update: I added this.

@albrow albrow force-pushed the feature/dynamic-max-expiration branch from 963aa94 to 4eded3b Oct 22, 2019
albrow and others added 12 commits Oct 15, 2019
Co-Authored-By: Fabio B <me@fabioberger.com>
Co-Authored-By: Fabio B <me@fabioberger.com>
albrow added 14 commits Oct 19, 2019
@albrow albrow force-pushed the feature/dynamic-max-expiration branch from ddae068 to a5bfe82 Oct 23, 2019
browser/ts/index.ts Outdated Show resolved Hide resolved
meshdb/meshdb_test.go Show resolved Hide resolved
zeroex/orderwatch/order_watcher.go Show resolved Hide resolved
zeroex/orderwatch/order_watcher.go Show resolved Hide resolved
zeroex/orderwatch/order_watcher.go Show resolved Hide resolved
zeroex/orderwatch/order_watcher.go Outdated Show resolved Hide resolved
// Final expiration time check before inserting the order. We might have just
// changed max expiration time above.
if orderInfo.SignedOrder.ExpirationTimeSeconds.Cmp(w.maxExpirationTime) == 1 {
return nil

This comment has been minimized.

Copy link
@fabioberger

fabioberger Oct 24, 2019

Contributor

Won't this mean that the node operator won't get the STOPPED_WATCHING event emitted?

This comment has been minimized.

Copy link
@albrow

albrow Oct 24, 2019

Author Member

@fabioberger Yes but it also means they never received the ADDED event, which seems consistent to me. What do you think is the right behavior here?

This comment has been minimized.

Copy link
@fabioberger

fabioberger Oct 24, 2019

Contributor

Since we ACKd the incoming JSON-RPC request, we should return STOPPED_WATCHING.

zeroex/orderwatch/order_watcher.go Show resolved Hide resolved
zeroex/orderwatch/order_watcher.go Outdated Show resolved Hide resolved
zeroex/orderwatch/slowcounter/slow_counter.go Outdated Show resolved Hide resolved
albrow and others added 6 commits Oct 24, 2019
Co-Authored-By: Fabio B <me@fabioberger.com>
Co-Authored-By: Fabio B <me@fabioberger.com>
Co-Authored-By: Fabio B <me@fabioberger.com>
…/0x-mesh into feature/dynamic-max-expiration
Copy link
Contributor

fabioberger left a comment

Small typo but otherwise LGTM!

zeroex/orderwatch/order_watcher.go Outdated Show resolved Hide resolved
albrow and others added 2 commits Oct 24, 2019
Co-Authored-By: Fabio B <me@fabioberger.com>
@albrow albrow merged commit 97c1127 into development Oct 24, 2019
1 check passed
1 check passed
ci/circleci: build Your tests passed on CircleCI!
Details
@albrow albrow deleted the feature/dynamic-max-expiration branch Oct 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.