add size-limiting to the DictDB backend #1930

Merged
merged 1 commit into from Jul 23, 2012

Conversation

Projects
None yet
3 participants
@minrk
Owner

minrk commented Jun 12, 2012

Adds configurables for culling old records when limits are exceeded.

Default limits: 1GB of buffers, 1024 records, 10% cull.

@fperez

This comment has been minimized.

Show comment Hide comment
@fperez

fperez Jun 12, 2012

Owner

Awesome! Tagged for 0.14.

Owner

fperez commented Jun 12, 2012

Awesome! Tagged for 0.14.

@bfroehle

This comment has been minimized.

Show comment Hide comment
@bfroehle

bfroehle Jun 12, 2012

Contributor

I haven't investigated this thoroughly, but are we handling (numpy) array buffers here?

If so, len(x) is not a useful metric for the size of the buffer.

>> import numpy as np
>>> x = np.ones((1,100000))
>>> len(x)
1
>>> x.size
100000

Also how does this handle buffers which may be shared between different objects?

>>> y = x[::2]
>>> y.base is x

(Apologies if I am totally mis-reading the intention of this pull request... Perhaps this is just storing picked versions of the objects?)

Contributor

bfroehle commented Jun 12, 2012

I haven't investigated this thoroughly, but are we handling (numpy) array buffers here?

If so, len(x) is not a useful metric for the size of the buffer.

>> import numpy as np
>>> x = np.ones((1,100000))
>>> len(x)
1
>>> x.size
100000

Also how does this handle buffers which may be shared between different objects?

>>> y = x[::2]
>>> y.base is x

(Apologies if I am totally mis-reading the intention of this pull request... Perhaps this is just storing picked versions of the objects?)

@minrk

This comment has been minimized.

Show comment Hide comment
@minrk

minrk Jun 12, 2012

Owner

The Hub only ever gets raw buffers from zeromq, no buffers can be shared, there are no Python objects involved (at least of type other than buffer/memoryview with element size=1).

Owner

minrk commented Jun 12, 2012

The Hub only ever gets raw buffers from zeromq, no buffers can be shared, there are no Python objects involved (at least of type other than buffer/memoryview with element size=1).

@fperez

This comment has been minimized.

Show comment Hide comment
@fperez

fperez Jun 13, 2012

Owner

Heads-up, in case you want to keep it mergeable, it has a conflict now. Those tend to be easiest to deal with if they don't accumulate.

Owner

fperez commented Jun 13, 2012

Heads-up, in case you want to keep it mergeable, it has a conflict now. Those tend to be easiest to deal with if they don't accumulate.

@minrk

This comment has been minimized.

Show comment Hide comment
@minrk

minrk Jun 13, 2012

Owner

indeed - my two db PRs touched one line of the same file. Rebased, thanks!

Owner

minrk commented Jun 13, 2012

indeed - my two db PRs touched one line of the same file. Rebased, thanks!

@minrk

This comment has been minimized.

Show comment Hide comment
@minrk

minrk Jul 21, 2012

Owner

Test results for commit 7a85cbc merged into master
Platform: darwin

  • python2.6: OK (libraries not available: cython matplotlib oct2py pygments pymongo qt rpy2 tornado wx wx.aui)
  • python2.7: OK (libraries not available: oct2py wx wx.aui)
  • python3.2: OK (libraries not available: cython matplotlib oct2py pymongo qt rpy2 wx wx.aui)

Not available for testing:

Owner

minrk commented Jul 21, 2012

Test results for commit 7a85cbc merged into master
Platform: darwin

  • python2.6: OK (libraries not available: cython matplotlib oct2py pygments pymongo qt rpy2 tornado wx wx.aui)
  • python2.7: OK (libraries not available: oct2py wx wx.aui)
  • python3.2: OK (libraries not available: cython matplotlib oct2py pymongo qt rpy2 wx wx.aui)

Not available for testing:

@minrk

This comment has been minimized.

Show comment Hide comment
@minrk

minrk Jul 23, 2012

Owner

Merging this, as it should be tested in the wild.

Owner

minrk commented Jul 23, 2012

Merging this, as it should be tested in the wild.

minrk added a commit that referenced this pull request Jul 23, 2012

Merge pull request #1930 from minrk/limitdictdb
add size-limiting to the DictDB backend

Adds configurables for culling old records when limits are exceeded.

Default limits: 1GB of buffers, 1024 records, 10% cull.

@minrk minrk merged commit e0803aa into ipython:master Jul 23, 2012

@minrk minrk deleted the minrk:limitdictdb branch Mar 31, 2014

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this pull request Nov 3, 2014

Merge pull request #1930 from minrk/limitdictdb
add size-limiting to the DictDB backend

Adds configurables for culling old records when limits are exceeded.

Default limits: 1GB of buffers, 1024 records, 10% cull.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment