Skip to content
Commits on Jul 8, 2010
  1. @steveyen @dustin

    bug 1366 - simplified moxi cmd-line

    ./moxi http://127.0.0.1:8080/pools/default/bucketsStreamingConfig/default
    
    and, all those -z and -Z cmd-line parameter overrides are still there.
    
    Also, updated the usage string, and updated the default listen ports
    to be more membase friendly.  The default listen ports now have
    define's for...
    
    MOXI_DEFAULT_LISTEN_PORT
    MEMCACHED_DEFAULT_LISTEN_PORT
    
    Change-Id: I4c3d2ce7ab86562b0d868891baa547ea3955a607
    Reviewed-on: http://review.northscale.com/976
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Dustin Sallings <dustin@spy.net>
    steveyen committed with dustin Jun 29, 2010
  2. @steveyen @dustin

    updated README

    Change-Id: Iae2a95f948ab172927bfc42cfe9dffffb64375f5
    Reviewed-on: http://review.northscale.com/1047
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Dustin Sallings <dustin@spy.net>
    steveyen committed with dustin Jul 6, 2010
Commits on Jul 7, 2010
  1. @steveyen

    binary-to-binary handling of not-my-vbucket responses

    Change-Id: Ib7c493f19d964fb48d8d29d1d69238a74e2ce755
    Reviewed-on: http://review.northscale.com/975
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 29, 2010
  2. @steveyen

    more printf's around a2b not-my-vbucket errors

    Change-Id: I67cb8c60f56c21f2b87dee5298fcd34d1a22ca33
    Reviewed-on: http://review.northscale.com/974
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 29, 2010
  3. @steveyen

    cproxy_on_new_config() broken into smaller helper funcs

    There was an embarassingly huge function, with more lines of code
    than should be excused.  cproxy_on_new_config() now has
    helper functions of cproxy_on_new_config_json() and
    cproxy_on_new_config_kvs().  This is part of the code pathway when
    libconflate realizes it has a new (re-)configuration.
    
    There be more monster functions here, to be slain another day.
    
    Change-Id: I51ab96a3b9c4d7bad9d7d66b56a730e39168ed9c
    Reviewed-on: http://review.northscale.com/973
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 29, 2010
  4. @steveyen

    cproxy_binary_uncork_cmds()

    In this change, the uncorking of quiet binary commands is finally
    implemented, with a proper refcounting of item structs.
    
    b2b_forward_item() and b2b_forward_item_vbucket() helper functions
    were refactored into existence to allow the uncorking code to easily
    "replay" all the corked commands.
    
    Along the way, having a vbucketId of -1 turned out to be meaningful in
    this commit where it means we really don't have a vbucketId, like a
    SQL NULL.  vbucketId of -1 occurs in the code when we're doing a
    non-key'ed broadcast command, like a NOOP or FLUSH_ALL.
    
    I also discovered we shouldn't clean up the corked commands
    linked-list during conn_cleanup(), but instead do the cleanup on
    conn_free().  It turns out conn_cleanup() is invoked every time
    through the drive_machine() loop when we move back to conn_new_cmd
    state, which made the code incorrectly forget all the corked quiet
    commands.
    
    Change-Id: I0d371a3a0c964e01cd4579de02c7244106275479
    Reviewed-on: http://review.northscale.com/944
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 28, 2010
  5. @steveyen

    binary-to-binary code refactoring & prep work to cork quiet cmds

    Binary proxying is now implemented in this change by allocating an
    appropriately sized item (sizeof(binary_header) + bodylen).  The
    ITEM_data() will hold the entire binary request header and body bytes
    (which will be conn_nread into the ITEM_data() area).  This allows a
    simpler implementation which handles both regular binary commands,
    forthcoming corked quiet commands (which will be completed in a future
    commit), and is also helpful to simplify the code that retries
    commands during not-my-vbucket errors (which you'll also see in a
    future commit).  In all those cases, the conn->item will have
    everything that moxi needs to transmit to the downstream memcached
    server(s) in one shot (an easy conn_mwrite).
    
    Along the way, there are more helper functions, and a new bin_cmd
    struct to record corked binary quiet commands.
    
    Quiet binary commands are corked until there's a non-quiet command.
    The corking just holds items a linked-list (bin_cmd struct's).  The
    head of the linked-list is on the upstream conn struct, so corking
    happens per-session.
    
    We using corking currently because moxi has so much dependence on
    memcached/drive_machine/libevent infrastructure, instead of a more
    flexible infrastructure (say, C++).
    
    cproxy_update_event_write() helped to reduce some code duplication,
    which just prepares a connection for libevent update_event().
    
    cproxy_dump_header() emits a binary header msg, making Trond's
    binary_header printf() code a little more reusable.
    
    In moxi, as a side note, VERSION and QUIT commands are handled as
    special-cases, in that these commands are not forwarded to the
    downstream memcached server(s).  QUIT, especially, does not make sense
    to proxy.
    
    Change-Id: Ieae3e018cc4fa18fc231839975a0b3f836e9b3cc
    Reviewed-on: http://review.northscale.com/943
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 27, 2010
  6. @steveyen

    one more assert

    and TODO's comments around binary-to-binary
    
    Change-Id: I53c487c46339eb327b98157ef7f7286764789840
    Reviewed-on: http://review.northscale.com/942
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 27, 2010
  7. @steveyen

    mock tests pass, by todo'ing the dedupe tests

    THe dedupe tests fail because the not-my-vbucket retry mechanism
    relies on the multi-key dedupe machinery, changing the dedupe
    feature behavior.
    
    Change-Id: I77c2a99d3eda419680c028c83f454468a8c9ab15
    Reviewed-on: http://review.northscale.com/928
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 27, 2010
  8. @steveyen

    passing ./t/whitespace.t

    And, missing a compilation warning in the previous check-in
    around an unused item *it variable.
    
    Change-Id: Ib4613e67688701f8020257fb4f3ed0d6e9f15082
    Reviewed-on: http://review.northscale.com/927
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 26, 2010
  9. @steveyen

    MOXI_USE_VBUCKET and MOXI_ITEM_MALLOC in configure.ac

    This is so that moxi can still be conditionally compiled with
    libmemcached, (libmemcached-0.40-revno839), instead of libvbucket, and
    compiled to use the classic memcached slabber (instead of malloc) for
    items.
    
    This should be temporary until the unit and mock tests are updated.
    
    Change-Id: I416bd1d54e52a0f56b225f1c0a851f9ae13b0700
    Reviewed-on: http://review.northscale.com/926
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 26, 2010
Commits on Jun 30, 2010
  1. @steveyen @seanlynch

    bug 1648 - need to set vbucket ID during single-key a2b commands

    This should fix any single-key simple (non-item) ascii commands
    (delete, incr/decr), including single-key ascii GET's.
    
    The code was incorrectly not putting the vbucket ID into the request
    opaque field.  During a not-my-vbucket response, the opaque field
    was always providing a vbucket ID that was incorrectly 0.
    
    Change-Id: I9ee65760685835cb83e941d03da94fa80f6353e8
    Reviewed-on: http://review.northscale.com/972
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Sean Lynch <seanl@literati.org>
    steveyen committed with seanlynch Jun 29, 2010
Commits on Jun 25, 2010
  1. @steveyen @dustin

    bug 1554 - printfs in moxi to help debug the missing data bug

    Change-Id: I2c57bf88ad9c03eb6c65de936ab398ea2d969cfc
    Reviewed-on: http://review.northscale.com/896
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    steveyen committed with dustin Jun 25, 2010
Commits on Jun 23, 2010
  1. @steveyen @dustin

    bug 1525 - moxi eating 100% cpu due to conn_pause

    Compared to memcached, moxi adds a new 'conn_pause' state to the
    drive_machine() state machine.  Some of the new membase/vbucket features,
    especially when encountering a pending vbucket (where the downstream
    server blocks for a potentially appreciably long time), were
    exercising this codepath a lot more.  In this case, the upstream
    connection (to the application/client) was left in conn_pause
    state.
    
    In one of the situations where I caught the 100% cpu situation, the
    drive_machine() was spin looping on conn_pause'ed connections.
    
    One thing that might lead to this is moxi code seems to (incorrectly)
    not be unregistering the libevent registrations for upstream
    connections that go into conn_pause state.
    
    As a catch-all fix, any connections that are in conn_pause will have
    their libevent registrations event_del()'ed.
    
    Change-Id: Ia88133a6cb97209fff6bab465a14b934b2dd6274
    Reviewed-on: http://review.northscale.com:8080/844
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Dustin Sallings <dustin@spy.net>
    steveyen committed with dustin Jun 22, 2010
Commits on Jun 22, 2010
  1. @steveyen

    bug 1512 - don't leak timeout_event registrations

    Previously, if the code encountered a not-my-vbucket error
    and wanted to retry the command, it would leak a libevent evtimer
    registration, which eventually led to moxi pegging 100% cpu
    with all the timer events.
    
    Change-Id: I3d966246713b2c095b17ff2ffacb9943e2b90c87
    Reviewed-on: http://review.northscale.com:8080/813
    Reviewed-by: Eric Lambert <eric.d.lambert@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 21, 2010
  2. @steveyen

    bug 1512 - use less deadlock prone pathway for simple cmd retries

    for simple ascii-2-binary commands (set, delete, single-key GET's),
    use the upstream_retry counter signalling approach rather
    than the fancy, more race-prone work queues.
    
    Change-Id: Id6b34fe5cbec102ca280094d1261dc5d86eeae30
    Reviewed-on: http://review.northscale.com:8080/812
    Reviewed-by: Eric Lambert <eric.d.lambert@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 21, 2010
  3. @steveyen

    better debug printfs

    Change-Id: Ibeef9d43ff729021505e94ecdb710e4902dafcba
    Reviewed-on: http://review.northscale.com:8080/811
    Reviewed-by: Eric Lambert <eric.d.lambert@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 21, 2010
Commits on Jun 21, 2010
  1. @steveyen @CaptTofu

    bug 1450 - optimization for ascii-2-binary single-key GET

    moxi already had multi-key GET machinery working for ascii-to-binary,
    which was the more complex case (by using GETKQ's and NOOP's).  This
    optimization now handles single-key ascii GET's with a codepath that
    just uses a single binary GETK message.
    
    Change-Id: Ia847073dd5ab5e401f87bf84238c8a0f50a788af
    Reviewed-on: http://review.northscale.com:8080/781
    Tested-by: Patrick M. Galbraith <patg@patg.net>
    Reviewed-by: Patrick M. Galbraith <patg@patg.net>
    steveyen committed with CaptTofu Jun 21, 2010
Commits on Jun 20, 2010
  1. @steveyen @CaptTofu

    key_stats_matcher doesn't have to be multithread safe

    since each key_stats_matcher is only scoped to a single worker thread,
    it doesn't need to be created with a multithreaded flag of true.
    
    Change-Id: I613011bfa04d7800352a32541730fad0ddb9dc9d
    Reviewed-on: http://review.northscale.com:8080/742
    Tested-by: Patrick M. Galbraith <patg@patg.net>
    Reviewed-by: Patrick M. Galbraith <patg@patg.net>
    steveyen committed with CaptTofu Jun 19, 2010
  2. @steveyen @CaptTofu

    emitting more error codes in ascii-to-binary

    Change-Id: Ic81f242be2a1e4b75d175a3efc4694b5d57eec07
    Reviewed-on: http://review.northscale.com:8080/743
    Tested-by: Patrick M. Galbraith <patg@patg.net>
    Reviewed-by: Patrick M. Galbraith <patg@patg.net>
    steveyen committed with CaptTofu Jun 20, 2010
Commits on Jun 19, 2010
  1. @steveyen

    bug 1429 - stop retrying after too many ascii-to-binary retries

    There are two cases of retry tracking added here, so code
    needed fixing in 2 different places...
    
    - for simple, single-shot mutation commands
    - for scatter-gather GETKQ's and NOOP translation case
    
    Also, fixed a case when the multiget was NULL, introduced
    in a previous performance enhancement fix.
    
    Change-Id: I043f12f9733624b225196c29b144f5b1e02e5852
    Reviewed-on: http://review.northscale.com:8080/700
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 19, 2010
  2. @steveyen

    turn off the front_cache matcher and key_stats by default

    for more performance
    
    Change-Id: Idcefbc13bed99952bbd8633f8057b2296301e8d9
    Reviewed-on: http://review.northscale.com:8080/697
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 19, 2010
  3. @steveyen

    moxi is faster without slabber

    Using the slabber's global lock is worse than using straight malloc in
    moxi.  This change probably prevents moxi from working right as a
    memcached server, but it'll now be a faster proxy, consuming less CPU
    resource.
    
    Change-Id: I976aa01c1c1a24d665d9cadcfd3e3c8c43fbf095
    Reviewed-on: http://review.northscale.com:8080/696
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Matt Ingenthron <matt@northscale.com>
    Reviewed-by: Eric Lambert <eric.d.lambert@gmail.com>
    steveyen committed Jun 19, 2010
  4. @steveyen

    avoid dedupe map during single-key gets

    Change-Id: I84f6ca20ae0b52bce59e543d7de405f2e00923f0
    Reviewed-on: http://review.northscale.com:8080/695
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Eric Lambert <eric.d.lambert@gmail.com>
    Reviewed-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 18, 2010
  5. @steveyen

    b2b upstream & downstream for simple, non-quiet commands

    Change-Id: I25c4e4a4fa1cac101aa19853f47857914dfc0836
    Reviewed-on: http://review.northscale.com:8080/687
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 15, 2010
  6. @steveyen

    negotiating_proxy_prot added

    Change-Id: I3891502bacdf0701813060548dc66f8927374492
    Reviewed-on: http://review.northscale.com:8080/686
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Eric Lambert <eric.d.lambert@gmail.com>
    steveyen committed Jun 15, 2010
  7. @steveyen

    binary protocol fix to is_compatible_request

    Change-Id: Ib7c9d034ceee29502409571c6f0d6f3c4b9030dc
    Reviewed-on: http://review.northscale.com:8080/685
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 15, 2010
  8. @steveyen @CaptTofu

    bug 1074 - upstream_error() supports b2b proxying

    Need to write a binary error response when the upstream protocol
    is binary protocol.
    
    Change-Id: Iacdb108b42abfe6cf57b8e028bd744dd2978a144
    Reviewed-on: http://review.northscale.com:8080/684
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Tested-by: Patrick M. Galbraith <patg@patg.net>
    Reviewed-by: Patrick M. Galbraith <patg@patg.net>
    steveyen committed with CaptTofu Jun 15, 2010
Commits on Jun 18, 2010
  1. @steveyen @dustin

    bug 1431 - moxi invokes vbucket_compare()

    so that moxi does a better job of reusing pre-existing
    downstream connections.
    
    Change-Id: I9f803b062bf6f94e7d9f0a52d3e697826de29a50
    Reviewed-on: http://review.northscale.com:8080/651
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Dustin Sallings <dustin@spy.net>
    steveyen committed with dustin Jun 17, 2010
  2. @steveyen

    upstream_retry when multi-GET not-my-vbucket err

    Change-Id: Iebbae35b0509cc16cbaf4610a3e433b60b71b94b
    
    In this implementation of handling not-my-bucket errors during
    ascii-to-binary proxying, we use the binary header's opaque field.
    
    moxi uses the opaque field to store either a vbucket id or a key
    index, depending on the command opcode.  The key index is for a char
    array, and is used to retreive the key for a given command.  Given the
    key, moxi can then re-retrieve the vbucket id.
    
    Given the vbucket id, finally, during a not-my-vbucket error, moxi can
    then call the incorrect_master() API of libvbucket.
    
    Change-Id: I8f7f015a4d0392834778654c2b9441f432a09db0
    Reviewed-on: http://review.northscale.com:8080/642
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Tested-by: Patrick M. Galbraith <patg@northscale.com>
    Reviewed-by: Patrick M. Galbraith <patg@northscale.com>
    steveyen committed Jun 17, 2010
Commits on Jun 17, 2010
  1. @steveyen @dustin

    hooked up vbucket_found_incorrect_master()

    Change-Id: I20c9da6f775a3532c7406430c261195a0f6b47b8
    Reviewed-on: http://review.northscale.com:8080/635
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Dustin Sallings <dustin@spy.net>
    steveyen committed with dustin Jun 17, 2010
  2. @steveyen @dustin

    handle not-my-vbucket error for simple, non-get, with-reply cmds

    Change-Id: I35a1cd83306d13bc6c80735c3a5fdda24f6d5a4f
    Reviewed-on: http://review.northscale.com:8080/634
    Tested-by: Dustin Sallings <dustin@spy.net>
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    steveyen committed with dustin Jun 17, 2010
  3. @steveyen @dustin

    updated moxi's PROTOCOL_BINARY enum's

    Change-Id: I724f287b0ba2b40c5936aa6c578d4af87b71a8f4
    Reviewed-on: http://review.northscale.com:8080/633
    Tested-by: Dustin Sallings <dustin@spy.net>
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    steveyen committed with dustin Jun 17, 2010
Commits on Jun 15, 2010
  1. @steveyen

    bug 1362 - more casting to allow rhel build

    Change-Id: I028108b704f015ccd958962d99c5aecd92ef4e66
    Reviewed-on: http://review.northscale.com:8080/576
    Reviewed-by: Eric Lambert <eric.d.lambert@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
    steveyen committed Jun 15, 2010
  2. @steveyen

    bug 1362 - moxi should use long long unsigned int

    Change-Id: I4b2a9f18dc2ac7ced7159a7cdbf808b41fa7842b
    Reviewed-on: http://review.northscale.com:8080/570
    Reviewed-by: Patrick M. Galbraith <patg@northscale.com>
    Tested-by: Patrick M. Galbraith <patg@northscale.com>
    steveyen committed with Patrick M. Galbraith Jun 15, 2010
Something went wrong with that request. Please try again.