Skip to content
Commits on Mar 25, 2013
  1. @abh1nay
  2. @abh1nay
  3. @abh1nay

    Updated release number

    abh1nay committed Mar 25, 2013
  4. @abh1nay

    Updated release notes

    abh1nay committed Mar 25, 2013
  5. @abh1nay
  6. @abh1nay

    Minor changes to test case# Please enter the commit message for your …

    …changes. Lines starting
    abh1nay committed Mar 25, 2013
Commits on Mar 22, 2013
  1. @abh1nay

    Added test case to test runtime exceptions

    Cleaned up the try catch  logic
    abh1nay committed Mar 22, 2013
  2. @abh1nay

    Adding test cases whch simulate intermittent exceptions etc

    in the hdfsfetcher and simulate retry logic
    this ensures checksum calculation is robust
    abh1nay committed Mar 21, 2013
  3. @abh1nay

    Adding an extra catch block for Exception and Throwable types. This i…

    …s used to catch the ClassNotFound exceptions
    Chinmay Soman committed with abh1nay Mar 21, 2013
  4. @abh1nay

    Using a per file checksum generator in the file copy in HdfsFetcher. …

    …This is used to handle the case where we might retry the copy in case of a Filesystem (hdfs) error.
    Chinmay Soman committed with abh1nay Mar 21, 2013
  5. @jayjwylie
  6. @jayjwylie
Commits on Mar 20, 2013
  1. @jayjwylie
  2. @jayjwylie
  3. @jayjwylie

    Addressed all code review comments for KeySampler and KeyVersionFetch…

    …er. Renamed many classes and methods related to FetchStreamRequestHandler.
    - All sub-classes of FetchStreamRequestHandler have been renamed to have a more consistent nomenclature.
    - Did some further refactoring in the FullScan* classes to move more work from leaf classes to
    - moved scan accounting to overall bae class
    - Added getNodesPartitionIdForKey method to StoreInstance to help with some fetch logic
    jayjwylie committed Mar 20, 2013
  4. @jayjwylie
  5. @jayjwylie

    Addressed review feedback and TODOs for KeyVersionSamplerCLI (and ren…

    …amed it to KeyVersionFetcherCLI).
    - mostly usability changes about command line options...
    - one copyright fix
    jayjwylie committed Mar 14, 2013
  6. @jayjwylie

    Addressed all review feedback and TODOs for KeySamplerCLI

    - added options: --store-names, --partition-ids, --keys-per-second-limit, and --progress-period-ops
    - got rid of unnecessary (and weird) retry loop. Can add seomthing like that later if needed.
    - pass all partitions to fetcher now instead of one-at-a-time
    Also did cosmetic fixes for KeyVersionSamplerCLI and
    jayjwylie committed Mar 14, 2013
  7. @jayjwylie

    Correctness fixes and significant refactoring of Fetch*StreamRequestH…

    …andlers. Expanded AdminFetchTest.
    Added more common helper methods to common base class of all fetchers FetchStreamRequestHandler.
    Added abstract base classes for partition-based fetching and non-partition-based fetching:
    - FetchPartitionStreamRequestHandler (partition-based)
    - FetchItemsStreamRequestHandler (non-partition-based)
    Refactored some code up to abstract base classes and made implementations as similar as possible (without heroic efforts) across all fetchers:
    - FetchEntriesStreamRequestHandler
    - FetchKeysStreamRequestHandler
    - FetchPartitionEntriesStreamRequestHandler
    - FetchPartitionKeysStreamRequestHandler
    Significant better test coverage in AdminFetchTest
    - tests fetching keys as well as fetching entries
    - tests partition-aware and non-partition-aware servers
    - tests per-partition limits on entries/keys fetched
    All of this clean up and additional testing lead to minor correctness fixes.
    Minor other clean ups of comments, override annotations, and fixes for KeySamplerCLI.
    jayjwylie committed Mar 14, 2013
  8. @jayjwylie

    change maxRecords to recordsPerPartition in fetch API and protobuf

    These are cosmetic changes. The client-side and server-side code does not properly do recordsPerPartition yet.
    Added a few TODOs in the code too.
    jayjwylie committed Mar 13, 2013
  9. @jayjwylie

    remove skipRecords from fetching API and protobuf

    AFAIK skipRecords was never used. By inspection, the code that would have been exercised if it had been used has never been correct. Removing skipRecords from the code base.
    - Added a number of TODOs to the code from the reviews
    - Changed some variable names
    jayjwylie committed Mar 13, 2013
  10. @jayjwylie
  11. @jayjwylie

    Minor fix for change to AdminClient

    - do not close down AdminStoreClient from queryKeys
    - added some additional checks to test to confirm (non)existence of exceptions&values
    jayjwylie committed Mar 11, 2013
  12. @jayjwylie

    Many minor tweaks to ConsistencyFix code and related files to address…

    … minor review feedback.
    - fixed commenting out of 'protobuff' target
    - add ClientConfig to constructor. This is needed for AdminStoreClient creation. It is confusing that we need both an AdminClientConfig and ClientConfig, but that is because the *ClientConfig code is so clumsy.
    - changed ".stop()" methods to ".close()" to be consistent with other interfaces.
    et cetera
    - Updated all copyright notices that have changed on this branch since December. This touched a ton of files...
    - annotated some TODOs with "(refactor)" to make refactoring todos easier to find.
    jayjwylie committed Mar 11, 2013
  13. @jayjwylie

    Added unit tests for ConsistencyFix, ConsistencyFixWorker, and QueryK…

    Many other fixes and cleanup:
    - tweak many variable names
    - add close method to stop adminClient
    - broke out BadKey to wrap a key with its string representation st failed fixes of badkey's can be dumped in full to file to be retried (without any additional effort)
    - marked 'parseVersion' as deprecated since, if we do this again, we should dump bytes not strings
    - track obsolete version exceptions and various statuses in Stats
    - clean up of arguments, variable names, etc.
    - cleanly close down fixer...
    - more logger.trace output
    - minor cleanup
    - added getVersioned() helper method
    - added consistency-fix store
    - marked all tests as @Test
    - update copyright notice
    jayjwylie committed Mar 7, 2013
  14. @jayjwylie
  15. @jayjwylie
  16. @jayjwylie

    Added KeySampler and KeyVersionSampler tools as a first step towards …

    …replacing "entropy" tool. Added another argument to bulk fetch operations that specifies maxRecords so that server can fetch a subset of a partition.
    - Samples keys from a cluster
    - Given file that lists keys per store, samples versions from each "responsible node" for that key
    - passed maxRecords through
    - TODO for future clean up of some types
    - auto generated!
    - white space
    - handle maxRecords
    - handle maxRecords
    - fixed usage of skipRecords
    - added maxRecords
    - added mac_records to protobuff definition
    - added maxRecords field to test
    jayjwylie committed Mar 7, 2013
  17. @jayjwylie

    Made rebalance --show-plan slightly more verbose and added yet anothe…

    …r analysis for cluster balance ("zone primary").
    - print out hostname within plan to make it easier to read (rather than having to lookup node ID)
    - calculate "zone primary" balance to understand which hosted partitions act as pseudo-master when zoned routing is used.
    jayjwylie committed Mar 4, 2013
  18. @jayjwylie

    Review and cleanup of consistency checker.

    - added required argument for an output file name for bad keys
    - changed Reporter to print out 'just the key' to the output file; it
      outputs more info at DEBUG level in general.
    - removed 'quiet' option
    - throw exceptions:
      - if # partitions differ across clusters
      - if replication factor is hinky
      - if isExpired encounters unknown type
    - main catches exceptions and fails fast
    - changed system.out debugging to logger.trace
    jayjwylie committed Feb 26, 2013
  19. @jayjwylie
  20. @jayjwylie

    Tweaked Rebalancer --output-dir again to better name interim metadata…

    … files for each batch.
    jayjwylie committed Feb 22, 2013
  21. @jayjwylie

    Changed Rebalancer --output-dir option to append numbers to each .xml…

    … file it outputs so that we have access to interim cluster configs.
    jayjwylie committed Feb 21, 2013
  22. @zhongjiewu @jayjwylie

    Refactored Consistency Check

    zhongjiewu committed with jayjwylie Feb 20, 2013
  23. @jayjwylie
Something went wrong with that request. Please try again.