Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Commits on Feb 3, 2012
  1. Code cleanup after review round 1

    Chinmay Soman authored
Commits on Jan 27, 2012
Commits on Jan 26, 2012
Commits on Jan 25, 2012
Commits on Jan 13, 2012
  1. Added dynamic throttle limit in storage service and corresponding Hdf…

    Chinmay Soman authored
    …sFetcher changes
Commits on Jan 5, 2012
  1. Added extra logging messages to HdfsFetcher

    Chinmay Soman authored
Commits on Dec 25, 2011
Commits on Nov 30, 2011
  1. Initial changes for RO bandwidth throttler

    Chinmay Soman authored
Commits on Jul 14, 2011
  1. @rsumbaly

    Java doc changes

    rsumbaly authored
Commits on May 25, 2011
  1. @rsumbaly
  2. @rsumbaly
Commits on May 24, 2011
  1. @rsumbaly

    a) Fixed NPE in Hadoop store builder reducer ( which works on a per b…

    rsumbaly authored
    …ucket basis ) (b) Added two flags in the admin store swapper to enable/disable the rollbacks
Commits on May 18, 2011
  1. @rsumbaly
Commits on May 10, 2011
  1. @rsumbaly
Commits on May 8, 2011
  1. @rsumbaly

    Moving around some unit tests + RO file transfer now only works on no…

    rsumbaly authored
    …n-mmaped files [ to avoid over-writing existing serving data ]
Commits on Apr 19, 2011
  1. @rsumbaly

    Changed the bucket nomenclature for RO Store. Now if a key maps to a …

    rsumbaly authored
    …secondary partition we don't bucket it into "{secondary_partition}_{replica_type}" bucket, but instead put it into "{master_partition}_{replica_type}". This helps us with (A) RO rebalancing (B) future plan to make RO Stores 3 phase - (i) fetch single copy (ii) migrate the copy (iii) swap
Commits on Apr 14, 2011
  1. @rsumbaly
Commits on Apr 12, 2011
  1. @rsumbaly
  2. @rsumbaly
Commits on Apr 5, 2011
  1. @rsumbaly
  2. @rsumbaly
Commits on Apr 2, 2011
  1. @rsumbaly
Commits on Mar 25, 2011
  1. @rsumbaly
Commits on Mar 24, 2011
  1. @rsumbaly
  2. @rsumbaly

    First version of diff generating algorithm

    rsumbaly authored
    - Generating the diff where X is iterator over old data chunk file, Y is iterator as it comes through MR
    - Y = Y-> next happens implicitly after every operation ( MR )
    
      __X__     __Y__
      k | v     k | v
      _____     _____
      0 | B     0 | A
      1 | D     1 | B
                2 | C
                3 | E
    
    if ( X.key > Y.key )
      emit +,<position_in_Y>,Y.value
    if ( X.key == Y.key )
      if ( X.value != Y.value )
         emit -,<position_in_Y>,X.value
         emit +,<position_in_Y>,Y.value
      X = X -> next
    if ( X.key < Y.key )
      emit -,<position_in_Y>,X.value
      X = X -> next
    if ( !X.hasNext() )
      emit +,<position_in_Y>,Y.value
    if ( !Y.hasNext() )
      emit -,<position_in_Y>,X.value
Commits on Mar 23, 2011
  1. @rsumbaly

    a) Changed back RO to use one reducer per chunk - Why? Finding diff a…

    rsumbaly authored
    …t one chunk level is simpler than at chunk set level
    
    b) Changed collision iterator to return key as well
    c) Loads of unit tests to test other wrapper data file chunk set code
Commits on Mar 21, 2011
  1. @rsumbaly

    - Decoupled the iterators from RO stores. Now works on a set of data …

    rsumbaly authored
    …chunk files
    
    - More unit tests for the iterators
    - Changed the RO v2 format now to { num,[ k_size, v_size, k, v] * }. Also added support for iterating at (a) individual k/v level (b) collided k/v level [ useful for finding the diff ]
Commits on Mar 19, 2011
  1. @rsumbaly
Commits on Mar 18, 2011
  1. @rsumbaly

    As an optimization to decrease the number of reducers, pushed the gen…

    rsumbaly authored
    …eration of all chunk set files to one reducer ( instead of having one reducer for every individual chunk set file ).
    
    This will also help in finding differences at a chunk set level.
  2. @rsumbaly

    Updated V2 of RO Stores - Now contains chunks of a per partition per …

    rsumbaly authored
    …replica type basis [ primary, secondary, tertiary...]. This results in many files but now has two advantages
    
    a) All primary based chunks will be hotter compared to other chunks - Locality!
    b) Rebalancing becomes a *little* simpler
Commits on Mar 17, 2011
  1. @rsumbaly
  2. @rsumbaly
Commits on Mar 16, 2011
  1. @rsumbaly

    Better fetch + swap for RO - If a fetch fails on some node, we delete…

    rsumbaly authored
    … the successful fetches ( to save space ). If a swap fails on some node, we rollback to the previous versions correctly
Commits on Feb 14, 2011
  1. @rsumbaly
Commits on Feb 11, 2011
  1. @rsumbaly

    Change the size of RO-key-bucket size from upper 4 bytes of md5(key) …

    rsumbaly authored
    …to 8 bytes - Decreases collisions dramatically
Something went wrong with that request. Please try again.