Pipeline vhosts #410
Pipeline vhosts #410
Conversation
mattbennett
force-pushed
the
pipeline-vhosts
branch
from
f800f6e
to
1a1ada7
Compare
|
could you rebase this on top of master please? |
mattbennett
force-pushed
the
pipeline-vhosts
branch
from
1a1ada7
to
e835979
Compare
| @@ -140,3 +140,65 @@ def find_free_port(host='127.0.0.1'): | |||
| port = sock.getsockname()[1] | |||
| sock.close() | |||
| return port | |||
|
|
|||
|
|
|||
| class ResourcePipeline(object): | |||
There was a problem hiding this comment.
do we see this being re-used? (if not, maybe move to conftest?)
There was a problem hiding this comment.
I would like to use it in nameko-sqlalchemy to speed up creation of database schemas.
nameko/testing/utils.py
Outdated
| def get(self): | ||
| return self.ready.get() | ||
|
|
||
| def discard(self, vhost): |
There was a problem hiding this comment.
if we're keeping this generic, maybe vhost -> resource or similar
nameko/testing/utils.py
Outdated
| while self.ready.qsize(): | ||
| unused = self.ready.get() | ||
| self.trash.put(unused) | ||
| eventlet.sleep() # allow create thread to run |
There was a problem hiding this comment.
how come we need this? (is it not enough to wait in wait below?)
There was a problem hiding this comment.
It's for the case where the create loop is blocked waiting to put an item into the ready queue (i.e. waiting for it to be less than maxsize).
unused = self.ready.get() unblocks the create thread but doesn't yield, so we exit the while loop here before giving it a chance to run. Once unblocked, the create thread deposits its item into the ready queue, and if we've already exited the loop, that final item isn't destroyed.
I've expanded the inline comment to hopefully explain
nameko/testing/pytest.py
Outdated
| ) | ||
| rabbit_manager.create_vhost(vhost) | ||
| rabbit_manager.set_vhost_permissions(vhost, username, '.*', '.*', '.*') | ||
| vhost = vhost_pipeline.get() |
There was a problem hiding this comment.
thought on using a contextmanager interface for get and discard as well?
| STOP = object() | ||
|
|
||
| def __init__(self, create, destroy, size=3): | ||
| if size == 0: |
There was a problem hiding this comment.
how did you pick the size?
There was a problem hiding this comment.
size=3? From trial and error in the vhost usage -- 3 resulted in the best speedup. It's a balance between having something available when you need it and the "waste" of destroying what's left unused at the end of the run.
There was a problem hiding this comment.
so do you think 3 is still a reasonable default for other use-cases? (or do we e.g. want to force the user to choose?)
There was a problem hiding this comment.
I prefer having a default rather than forcing the user to select something. I think 3 is fine.
nameko/testing/utils.py
Outdated
| # allow create thread to run if it is blocked waiting for capacity | ||
| # in the ready queue. otherwise we'll exit this loop before the | ||
| # final item is added to the ready queue and it won't be destroyed | ||
| eventlet.sleep() |
There was a problem hiding this comment.
i'm still thinking about this but we might have a race here (e.g. if create is slow). may need e.g. a semaphore around the body of the while look in _create
There was a problem hiding this comment.
I think you're right. I'll rework it.
No description provided.