Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Commits on Jul 23, 2010
  1. Steve Yen

    work-in-progress on stats merging

    steveyen authored
    Change-Id: I19e0bff4c5a1105081d3efd4eccc54e1b392c96b
  2. Patrick Galbraith Steve Yen

    Added fixes for 'make dist' to work properly

    CaptTofu authored steveyen committed
Commits on Jul 22, 2010
  1. Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) Steve Yen

    fixed off-by-one in cproxy_stats

    alk authored steveyen committed
    The issue was found & reported by valgrind.
    
    Change-Id: I510ff330d861d1d8452675963444a310cfc131d5
    Reviewed-on: http://review.northscale.com/1288
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Steve Yen <steve.yen@gmail.com>
Commits on Jul 20, 2010
  1. Steve Yen Aliaksey Kandratsenka (aka Aliaksei Kandratsenka)

    bug 1825 - binary FLUSHQ needs conn_closing

    steveyen authored alk committed
    While bouncing client connections during binary-to-binary FLUSHQ, we
    avoid cproxy_close_conn() (which was meant to close conn_pause'd
    connections) and just use a straight conn_set_state(c, conn_closing).
    This matches more closely how the memcached networking machinery
    closes upstream connections and avoids a drive_machine() recursion,
    too.
    
    Change-Id: I962acd2d1b9eed4601b3bc1dbe7e8a88158d04c8
    Reviewed-on: http://review.northscale.com/1262
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  2. Steve Yen Aliaksey Kandratsenka (aka Aliaksei Kandratsenka)

    bug 1825 - binary FLUSHQ turned off for binary-to-binary

    steveyen authored alk committed
    Binary-to-binary FLUSHQ support requires a rethink.  Until then,
    turning off support for FLUSHQ so that it doesn't mess up
    the state of the downstream connections and put moxi into
    a weird state.
    
    Change-Id: I962acd2d1b9eed4601b3bc1dbe7e8a881584d0c8
    Reviewed-on: http://review.northscale.com/1261
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Commits on Jul 19, 2010
  1. Steve Yen Aliaksey Kandratsenka (aka Aliaksei Kandratsenka)

    bug 1807 - refcount overdecrement during binary-to-binary quiet cmds

    steveyen authored alk committed
    Change-Id: I4bf942096f3f858cfe13ee3c2b2ce9fbdb107b1b
    Reviewed-on: http://review.northscale.com/1259
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  2. Steve Yen

    bug 1497 - binary-2-binary broadcast command impl was wrong

    steveyen authored
    Before, the code didn't correctly handle a cluster situation for
    broadcast commands and stats merging was wrong for binary-to-binary
    commands.
    
    With this commit, cluster-wide binary-to-binary broadcast commands
    (like NO-OP, FLUSH, and STATS) utilize the same mechanism like
    ascii-2-binary broadcast commands.
    
    That is, there is an upstream_suffix set of bytes associated with each
    client broadcast command.  This upstream_suffix is the last response
    message that is sent to back to the upstream client after all replies
    have been gathered back from the downstream servers.
    
    In the binary-to-binary case, this upstream_suffix would be a NO-OP,
    FLUSH, or STATS response.  (In the ascii, case, the upstream_suffix
    would be something like "END", or "OK").
    
    The upstream_suffix was extended to also have an optional, explicit
    upstream_suffix_len, which the code will use in the binary protocol
    case instead of doing a strlen(upstream_suffix).
    
    Finally, binary-to-binary stats was not doing the stats merging at all.
    That should admittedly be a separate commit.
    
    Change-Id: I9d105aa402c7e9f2c930aa3f51226ee7d569b539
    Reviewed-on: http://review.northscale.com/1256
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  3. Steve Yen

    bug 1813 - moxi exits if port is taken already

    steveyen authored Matt Ingenthron committed
    Change-Id: I06fb2459d5a2e7b2d5c7458eaaad85b0367e4aaa
    Reviewed-on: http://review.northscale.com/1246
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Matt Ingenthron <matt@northscale.com>
    Reviewed-by: Matt Ingenthron <matt@northscale.com>
  4. Steve Yen

    bug 1685 - added stats

    steveyen authored
    With this commit, we can see if any upstream connections are stuck in
    conn_pause state.  The new stats include...
    
     tot_retry_vbucket          - # retries due to not-my-vbucket errs
     tot_upstream_conn_paused   - when entering the conn_pause state
     tot_upstream_conn_unpaused - when leaving the conn_pause state
    
    tot_upstream_conn_paused should be only slightly greater than
    tot_upstream_conn_upaused during load, and should be the same when
    the cluster is quiescent.
    
    Change-Id: I53697db09fab89349c7d5deac837a1a6e99ff486
    Reviewed-on: http://review.northscale.com/1237
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  5. Steve Yen

    old TODO comment removed

    steveyen authored
    Change-Id: I8116a2706a153197bc68752035a0472fe1e6b720
    Reviewed-on: http://review.northscale.com/1236
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  6. Steve Yen

    python mc_bin_client/memcacheConstants.py for tests

    steveyen authored
    Change-Id: I61a330520e27e9767578735f7abf1796c42aaa6e
    Reviewed-on: http://review.northscale.com/1235
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  7. Steve Yen

    bug 1497 - setting OPAQUE_IGNORE_REPLY for simple ascii mutation cmds

    steveyen authored
    The OPAQUE_IGNORE_REPLY value was getting correctly marked for item-based
    mutation ascii noreply cmds (like set, prepend/append, add/replace), but
    incorrectly not for simple ascii noreply mutation cmds (like delete, incr/decr).
    
    This commit refactors out a helper function (a2b_set_opaque) to fix that.
    
    Change-Id: Icb7f43b729774abdc6d2e6a18bd0af85c83e5452
    Reviewed-on: http://review.northscale.com/1231
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  8. Steve Yen

    TODO comment and better printf

    steveyen authored
    Change-Id: I6332c62744861d61d0235c1ec8a09207a68d608d
    Reviewed-on: http://review.northscale.com/1230
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  9. Steve Yen

    bug 1792 - handle OPAQUE_IGNORE_REPLY for both ascii and binary clients

    steveyen authored
    This commit is with regards to handling ascii and binary clients concurrently.
    
    A client on one protocol (binary), might be assigned to use a
    downstream connection that was previously servicing another protocol
    (ascii).  The a2b protocol handler would use a special opaque magic
    value OPAQUE_IGNORE_REPLY to signal that a given binary response
    should be ignored.  This OPAQUE_IGNORE_REPLAY was used to map ascii
    "noreply" commands to binary quiet commands.
    
    This commit allows the b2b (binary-to-binary) codepath to handle any
    OPAQUE_IGNORE_REPLY which it encounters.  These responses might be
    sitting on the connection due to a2b (ascii-to-binary) usage.
    
    cproxy_binary_ignore_reply()
    
    Change-Id: Ie0232c3c72268d352d2ed553283b2abab6db7874
    Reviewed-on: http://review.northscale.com/1229
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
Commits on Jul 18, 2010
  1. Steve Yen

    bug 1783 - key pointer was wrong for hashing

    steveyen authored
    Change-Id: I1bbcb641802a2971fc023faac59a29c7734c117c
    Reviewed-on: http://review.northscale.com/1216
    Reviewed-by: Matt Ingenthron <matt@northscale.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
Commits on Jul 15, 2010
  1. Steve Yen

    moxi tests pass again

    steveyen authored
    With this change, the old moxi 'good tests' inherited from memcached
    pass again, with the testing topologies updated for libvbucket (the
    new default hashlib).  The change still allows the previous
    libmemcached hashlib as an option.
    
    To run the tests, use...
    
      ./t/moxi_all.pl
    
    And that's also part of 'make test' now, with better exit code
    propagation.
    
    Change-Id: Ie9bdd82170825f731efaa41ef1624babfcd17743
    Reviewed-on: http://review.northscale.com/1162
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  2. Steve Yen

    moxi testapp.c skips test_issue_72

    steveyen authored
    For now, moxi doesn't handle the MEMCACHED_PORT_FILENAME startup
    codepath, which is needed for test_issue_72 inherited from memcached.
    
    Change-Id: I09e37c358fa4a33917a2c66383422674755e0559
    Reviewed-on: http://review.northscale.com/1161
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  3. Steve Yen

    bug 1709 - double-free when using MOXI_ITEM_MALLOC

    steveyen authored
    The key is that assoc_insert() needs to own its own refcount.
    
    From Manik Taneja...
    
    I found a double free when moxi is run without the –z option.
    
    do_store_item()->do_item_link()->item_link()
    
    The item is subsequently freed through
    complete_nread_ascii()->item_remove(c->item). Since the item still exists in
    the linked list, a subsequent get for the same key will endup accessing the
    same item in the linked list and moxi tries to double free it in function
    complete_nread_ascii.
    
    Change-Id: I925aa2ce36ecdc322ffe63a827245c2cf40b73e1
    Reviewed-on: http://review.northscale.com/1150
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Manik Taneja <mtaneja@zynga.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Steve Yen <steve.yen@gmail.com>
Commits on Jul 14, 2010
  1. Steve Yen Aliaksey Kandratsenka (aka Aliaksei Kandratsenka)

    bug 1710: -Y cmd-line flag to allow control of stdin checking

    steveyen authored alk committed
    Found by Manik Taneja.
    
    Use the following to have moxi exit when stdin closes...
    
      ./moxi -Y y [...other flags...]
    
    With this change, by default, moxi no longer watches its stdin.
    
    Change-Id: Ia080366cc8f5d43f57d1416c24d21f40e908b88b
    Reviewed-on: http://review.northscale.com/1151
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Commits on Jul 8, 2010
  1. Steve Yen Dustin Sallings

    bug 1366 - simplified moxi cmd-line

    steveyen authored dustin committed
    ./moxi http://127.0.0.1:8080/pools/default/bucketsStreamingConfig/default
    
    and, all those -z and -Z cmd-line parameter overrides are still there.
    
    Also, updated the usage string, and updated the default listen ports
    to be more membase friendly.  The default listen ports now have
    define's for...
    
    MOXI_DEFAULT_LISTEN_PORT
    MEMCACHED_DEFAULT_LISTEN_PORT
    
    Change-Id: I4c3d2ce7ab86562b0d868891baa547ea3955a607
    Reviewed-on: http://review.northscale.com/976
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Dustin Sallings <dustin@spy.net>
  2. Steve Yen Dustin Sallings

    updated README

    steveyen authored dustin committed
    Change-Id: Iae2a95f948ab172927bfc42cfe9dffffb64375f5
    Reviewed-on: http://review.northscale.com/1047
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Dustin Sallings <dustin@spy.net>
Commits on Jul 7, 2010
  1. Steve Yen

    binary-to-binary handling of not-my-vbucket responses

    steveyen authored
    Change-Id: Ib7c493f19d964fb48d8d29d1d69238a74e2ce755
    Reviewed-on: http://review.northscale.com/975
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  2. Steve Yen

    more printf's around a2b not-my-vbucket errors

    steveyen authored
    Change-Id: I67cb8c60f56c21f2b87dee5298fcd34d1a22ca33
    Reviewed-on: http://review.northscale.com/974
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  3. Steve Yen

    cproxy_on_new_config() broken into smaller helper funcs

    steveyen authored
    There was an embarassingly huge function, with more lines of code
    than should be excused.  cproxy_on_new_config() now has
    helper functions of cproxy_on_new_config_json() and
    cproxy_on_new_config_kvs().  This is part of the code pathway when
    libconflate realizes it has a new (re-)configuration.
    
    There be more monster functions here, to be slain another day.
    
    Change-Id: I51ab96a3b9c4d7bad9d7d66b56a730e39168ed9c
    Reviewed-on: http://review.northscale.com/973
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  4. Steve Yen

    cproxy_binary_uncork_cmds()

    steveyen authored
    In this change, the uncorking of quiet binary commands is finally
    implemented, with a proper refcounting of item structs.
    
    b2b_forward_item() and b2b_forward_item_vbucket() helper functions
    were refactored into existence to allow the uncorking code to easily
    "replay" all the corked commands.
    
    Along the way, having a vbucketId of -1 turned out to be meaningful in
    this commit where it means we really don't have a vbucketId, like a
    SQL NULL.  vbucketId of -1 occurs in the code when we're doing a
    non-key'ed broadcast command, like a NOOP or FLUSH_ALL.
    
    I also discovered we shouldn't clean up the corked commands
    linked-list during conn_cleanup(), but instead do the cleanup on
    conn_free().  It turns out conn_cleanup() is invoked every time
    through the drive_machine() loop when we move back to conn_new_cmd
    state, which made the code incorrectly forget all the corked quiet
    commands.
    
    Change-Id: I0d371a3a0c964e01cd4579de02c7244106275479
    Reviewed-on: http://review.northscale.com/944
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  5. Steve Yen

    binary-to-binary code refactoring & prep work to cork quiet cmds

    steveyen authored
    Binary proxying is now implemented in this change by allocating an
    appropriately sized item (sizeof(binary_header) + bodylen).  The
    ITEM_data() will hold the entire binary request header and body bytes
    (which will be conn_nread into the ITEM_data() area).  This allows a
    simpler implementation which handles both regular binary commands,
    forthcoming corked quiet commands (which will be completed in a future
    commit), and is also helpful to simplify the code that retries
    commands during not-my-vbucket errors (which you'll also see in a
    future commit).  In all those cases, the conn->item will have
    everything that moxi needs to transmit to the downstream memcached
    server(s) in one shot (an easy conn_mwrite).
    
    Along the way, there are more helper functions, and a new bin_cmd
    struct to record corked binary quiet commands.
    
    Quiet binary commands are corked until there's a non-quiet command.
    The corking just holds items a linked-list (bin_cmd struct's).  The
    head of the linked-list is on the upstream conn struct, so corking
    happens per-session.
    
    We using corking currently because moxi has so much dependence on
    memcached/drive_machine/libevent infrastructure, instead of a more
    flexible infrastructure (say, C++).
    
    cproxy_update_event_write() helped to reduce some code duplication,
    which just prepares a connection for libevent update_event().
    
    cproxy_dump_header() emits a binary header msg, making Trond's
    binary_header printf() code a little more reusable.
    
    In moxi, as a side note, VERSION and QUIT commands are handled as
    special-cases, in that these commands are not forwarded to the
    downstream memcached server(s).  QUIT, especially, does not make sense
    to proxy.
    
    Change-Id: Ieae3e018cc4fa18fc231839975a0b3f836e9b3cc
    Reviewed-on: http://review.northscale.com/943
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  6. Steve Yen

    one more assert

    steveyen authored
    and TODO's comments around binary-to-binary
    
    Change-Id: I53c487c46339eb327b98157ef7f7286764789840
    Reviewed-on: http://review.northscale.com/942
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  7. Steve Yen

    mock tests pass, by todo'ing the dedupe tests

    steveyen authored
    THe dedupe tests fail because the not-my-vbucket retry mechanism
    relies on the multi-key dedupe machinery, changing the dedupe
    feature behavior.
    
    Change-Id: I77c2a99d3eda419680c028c83f454468a8c9ab15
    Reviewed-on: http://review.northscale.com/928
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  8. Steve Yen

    passing ./t/whitespace.t

    steveyen authored
    And, missing a compilation warning in the previous check-in
    around an unused item *it variable.
    
    Change-Id: Ib4613e67688701f8020257fb4f3ed0d6e9f15082
    Reviewed-on: http://review.northscale.com/927
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  9. Steve Yen

    MOXI_USE_VBUCKET and MOXI_ITEM_MALLOC in configure.ac

    steveyen authored
    This is so that moxi can still be conditionally compiled with
    libmemcached, (libmemcached-0.40-revno839), instead of libvbucket, and
    compiled to use the classic memcached slabber (instead of malloc) for
    items.
    
    This should be temporary until the unit and mock tests are updated.
    
    Change-Id: I416bd1d54e52a0f56b225f1c0a851f9ae13b0700
    Reviewed-on: http://review.northscale.com/926
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Steve Yen <steve.yen@gmail.com>
Commits on Jun 30, 2010
  1. Steve Yen Sean Richard Lynch

    bug 1648 - need to set vbucket ID during single-key a2b commands

    steveyen authored seanlynch committed
    This should fix any single-key simple (non-item) ascii commands
    (delete, incr/decr), including single-key ascii GET's.
    
    The code was incorrectly not putting the vbucket ID into the request
    opaque field.  During a not-my-vbucket response, the opaque field
    was always providing a vbucket ID that was incorrectly 0.
    
    Change-Id: I9ee65760685835cb83e941d03da94fa80f6353e8
    Reviewed-on: http://review.northscale.com/972
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Sean Lynch <seanl@literati.org>
Commits on Jun 25, 2010
  1. Steve Yen Dustin Sallings

    bug 1554 - printfs in moxi to help debug the missing data bug

    steveyen authored dustin committed
    Change-Id: I2c57bf88ad9c03eb6c65de936ab398ea2d969cfc
    Reviewed-on: http://review.northscale.com/896
    Tested-by: Steve Yen <steve.yen@gmail.com>
    Reviewed-by: Dustin Sallings <dustin@spy.net>
Commits on Jun 23, 2010
  1. Steve Yen Dustin Sallings

    bug 1525 - moxi eating 100% cpu due to conn_pause

    steveyen authored dustin committed
    Compared to memcached, moxi adds a new 'conn_pause' state to the
    drive_machine() state machine.  Some of the new membase/vbucket features,
    especially when encountering a pending vbucket (where the downstream
    server blocks for a potentially appreciably long time), were
    exercising this codepath a lot more.  In this case, the upstream
    connection (to the application/client) was left in conn_pause
    state.
    
    In one of the situations where I caught the 100% cpu situation, the
    drive_machine() was spin looping on conn_pause'ed connections.
    
    One thing that might lead to this is moxi code seems to (incorrectly)
    not be unregistering the libevent registrations for upstream
    connections that go into conn_pause state.
    
    As a catch-all fix, any connections that are in conn_pause will have
    their libevent registrations event_del()'ed.
    
    Change-Id: Ia88133a6cb97209fff6bab465a14b934b2dd6274
    Reviewed-on: http://review.northscale.com:8080/844
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Dustin Sallings <dustin@spy.net>
Commits on Jun 22, 2010
  1. Steve Yen

    bug 1512 - don't leak timeout_event registrations

    steveyen authored
    Previously, if the code encountered a not-my-vbucket error
    and wanted to retry the command, it would leak a libevent evtimer
    registration, which eventually led to moxi pegging 100% cpu
    with all the timer events.
    
    Change-Id: I3d966246713b2c095b17ff2ffacb9943e2b90c87
    Reviewed-on: http://review.northscale.com:8080/813
    Reviewed-by: Eric Lambert <eric.d.lambert@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
  2. Steve Yen

    bug 1512 - use less deadlock prone pathway for simple cmd retries

    steveyen authored
    for simple ascii-2-binary commands (set, delete, single-key GET's),
    use the upstream_retry counter signalling approach rather
    than the fancy, more race-prone work queues.
    
    Change-Id: Id6b34fe5cbec102ca280094d1261dc5d86eeae30
    Reviewed-on: http://review.northscale.com:8080/812
    Reviewed-by: Eric Lambert <eric.d.lambert@gmail.com>
    Tested-by: Steve Yen <steve.yen@gmail.com>
Something went wrong with that request. Please try again.