Skip to content

jackdoe/0XBAD

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

* 0XBAD broadcasty distributed shared memory key/value cache

  the whole thing is horribly simple, using FNV32 hashing to build a table 
  (without collision detection - so if 2 keys collide they are overwritten every time)

  [box A] - 
      my $v = $cache->find($key) || 
              $cache->store_and_broadcast($key,slow_subroutine(),0,12345);
              
  [box B] - daemon receives the broadcast message; adds it into the local cache
  [box C] - daemon receives the broadcast message; adds it into the local cache
  [box D] - udp is lost

  in this case, when a request for $key hits [box D] it will do the same 
  $cache->store_and_broadcast() and rebroadcast the item.

* install
  ruby: gem install xbad
  perl: cd perl && perl Makefile.PL && make && make test && make install

* usage
  my $CACHE = BAD->new(0xbeef,16,128)
    creates new shared memory segment with id 0xBEEF
    the segment size is 2^16 * 128 bytes, and it can hold at most 2^16 items

  my $value = $CACHE->find($key) - looks up
  my $value = $CACHE->store($key,$x_value,$expire_after);
  my $value = 
    $CACHE->store_and_broadcast($key,$x_value,$expire_after,$broadcast_port);

  if $expire_after is 0, the item never expires

  ruby: 
    b = XBAD.new(0xBAD,16,128)
    b.store(key,value,expire_after)
    value = b.find(key)

* broadcasting
  store_and_broadcast sends udp broadcast to the specified port
  
* fixed size
  every item in the cache is with size sp->item_size (specified on 
  pool creation) and the shared memory segment is created when you 
  specify exactly how many items between 2^16 and 2^32 you want to
  have in the pool.

  my $b = BAD->new(0xBEEF,17,128); 
      # will create 2^17 items with size 128 bytes

  this is very ineficient, but for caching small-ish items it works fine.

* speed

perl:
  (key: aaaaaaaaaaaaaaaabbbbbbbbbbbbbbb value 'a' x 100)
    native hash store: 4644160.73/s
          0xBAD store: 3062807.05/s
      memcached store: 24514.01/s

      native hash get: 3476691.62/s
            0xBAD get: 2153601.64/s
        memcached get: 25216.77/s

  * native hash - perl's $hash{$key} = $value and $value = $hash{$key}

ruby:
            hash set  3173242.4 i/s -   15841482 in   4.993171s
            xbad set  3760115.4 i/s -   18829228 in   5.008079s
            hash get  4008048.8 i/s -   20056464 in   5.004359s
            xbad get  1844151.9 i/s -    8363250 in   5.010949s
            hash mis  4262632.4 i/s -   21317955 in   5.001340s
            xbad mis  4441230.5 i/s -   22199103 in   4.999691s
        memcache set    13164.1 i/s -      66689 in   5.070756s
        memcache get    14114.6 i/s -      70788 in   5.026147s
       memcache cset    12767.4 i/s -      64501 in   5.076763s
       memcache cget    13474.4 i/s -      68202 in   5.082600s
      hash clone set   437421.6 i/s -    2196828 in   5.031107s
      hash clone get   446848.6 i/s -    2230452 in   4.996158s
    xbad set-encoded   794074.4 i/s -    3953088 in   5.006689s
    xbad get-encoded   372012.9 i/s -    1855488 in   5.035192s
 xbad set-encoded-sn   558399.0 i/s -    2798724 in   5.023483s
 xbad get-encoded-sn   372813.0 i/s -    1858272 in   5.028326s


  * hash- is ruby's native hash hash[key] = value and value = hash[key]
  * hash clone is hash[key] = value.clone and value = hash[key].clone
   xbad - encoded is Sereal (https://github.com/Sereal/Sereal) encoded
   xbad encoded-sn is Sereal encoded and snappy compressed

* locking
  everything is protected by simple gcc atomic builtins
  worst case is somebody locks an item and goes away (interrupted by signal
  or something) 

About

0XBAD - broadcasty distributed shared memory key/value cache

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published