Switch branches/tags
vector-clock-fixes v1.4.0 release-1.10.23-cutoff release-1.10.22-cutoff release-1.10.21-cutoff release-1.10.20-cutoff release- release-1.10.19-cutoff release-1.10.18-cutoff release-1.10.17-cutoff release-1.10.16-cutoff release-1.10.15-cutoff release-1.10.14-cutoff release-1.10.13-cutoff release-1.10.12-cutoff release-1.10.11-cutoff release-1.10.10-cutoff release-1.10.9-cutoff release-1.10.8-cutoff release-1.10.7-cutoff release-1.10.6-cutoff release-1.10.5-cutoff release-1.10.4-cutoff release-1.10.3-cutoff release-1.10.2-cutoff release-1.10.1-cutoff release-1.10.0-cutoff release-1.9.22-cutoff release-1.9.21-cutoff release-1.9.20-cutoff release-1.9.19-cutoff release-1.9.18-cutoff release-1.9.17-cutoff release-1.9.16-cutoff release-1.9.15-cutoff release-1.9.14-cutoff release-1.9.13-cutoff release-1.9.12-cutoff release-1.9.11-cutoff release-1.9.10-cutoff release-1.9.9-cutoff release-1.9.8-cutoff release-1.9.7-cutoff release-1.9.6-cutoff release-1.9.5-cutoff release-1.9.4-cutoff release-1.9.3-cutoff release-1.9.2-cutoff release-1.9.1-cutoff release-1.9.0-cutoff release-1.8.16-cutoff release-1.8.15-cutoff release-1.8.14-cutoff release-1.8.13-cutoff release-1.8.12-cutoff release-1.8.11-cutoff release-1.8.10-cutoff release-1.8.9-cutoff release-1.8.8-cutoff release-1.8.5-cutoff release-1.8.4-cutoff release-1.8.3-cutoff release-1.8.1-cutoff release-1.7.3-cutoff release-1.7.2-cutoff release-1.7.1-cutoff release-1.7.0-cutoff release-1.6.8-cutoff release-1.6.6-cutoff release-1.6.4 release-1.6.4-cutoff release-1.6.3-cutoff release-1.6.2-cutoff release-1.6.1-cutoff release-1.6.0-cutoff release-1.5.9-cutoff release-1.5.8-cutoff release-1.5.7-cutoff release-1.5.4-cutoff release-1.5.2-cutoff release-1.5.1-cutoff release-1.3.0-cutoff before-replicatype-was-removed before-donorbased-was-removed
Nothing to show
Commits on Mar 22, 2013
  1. Added test case to test runtime exceptions

    Cleaned up the try catch  logic
    abh1nay committed Mar 22, 2013
  2. Adding test cases whch simulate intermittent exceptions etc

    in the hdfsfetcher and simulate retry logic
    this ensures checksum calculation is robust
    abh1nay committed Mar 22, 2013
Commits on Mar 21, 2013
  1. Adding an extra catch block for Exception and Throwable types. This i…

    …s used to catch the ClassNotFound exceptions
    Chinmay Soman committed Mar 21, 2013
  2. Using a per file checksum generator in the file copy in HdfsFetcher. …

    …This is used to handle the case where we might retry the copy in case of a Filesystem (hdfs) error.
    Chinmay Soman committed Mar 21, 2013
Commits on Mar 20, 2013
  1. Addressed all code review comments for KeySampler and KeyVersionFetch…

    …er. Renamed many classes and methods related to FetchStreamRequestHandler.
    - All sub-classes of FetchStreamRequestHandler have been renamed to have a more consistent nomenclature.
    - Did some further refactoring in the FullScan* classes to move more work from leaf classes to
    - moved scan accounting to overall bae class
    - Added getNodesPartitionIdForKey method to StoreInstance to help with some fetch logic
    jayjwylie committed Mar 20, 2013
  2. Addressed review feedback and TODOs for KeyVersionSamplerCLI (and ren…

    …amed it to KeyVersionFetcherCLI).
    - mostly usability changes about command line options...
    - one copyright fix
    jayjwylie committed Mar 14, 2013
  3. Addressed all review feedback and TODOs for KeySamplerCLI

    - added options: --store-names, --partition-ids, --keys-per-second-limit, and --progress-period-ops
    - got rid of unnecessary (and weird) retry loop. Can add seomthing like that later if needed.
    - pass all partitions to fetcher now instead of one-at-a-time
    Also did cosmetic fixes for KeyVersionSamplerCLI and
    jayjwylie committed Mar 14, 2013
  4. Correctness fixes and significant refactoring of Fetch*StreamRequestH…

    …andlers. Expanded AdminFetchTest.
    Added more common helper methods to common base class of all fetchers FetchStreamRequestHandler.
    Added abstract base classes for partition-based fetching and non-partition-based fetching:
    - FetchPartitionStreamRequestHandler (partition-based)
    - FetchItemsStreamRequestHandler (non-partition-based)
    Refactored some code up to abstract base classes and made implementations as similar as possible (without heroic efforts) across all fetchers:
    - FetchEntriesStreamRequestHandler
    - FetchKeysStreamRequestHandler
    - FetchPartitionEntriesStreamRequestHandler
    - FetchPartitionKeysStreamRequestHandler
    Significant better test coverage in AdminFetchTest
    - tests fetching keys as well as fetching entries
    - tests partition-aware and non-partition-aware servers
    - tests per-partition limits on entries/keys fetched
    All of this clean up and additional testing lead to minor correctness fixes.
    Minor other clean ups of comments, override annotations, and fixes for KeySamplerCLI.
    jayjwylie committed Mar 14, 2013
  5. change maxRecords to recordsPerPartition in fetch API and protobuf

    These are cosmetic changes. The client-side and server-side code does not properly do recordsPerPartition yet.
    Added a few TODOs in the code too.
    jayjwylie committed Mar 13, 2013
  6. remove skipRecords from fetching API and protobuf

    AFAIK skipRecords was never used. By inspection, the code that would have been exercised if it had been used has never been correct. Removing skipRecords from the code base.
    - Added a number of TODOs to the code from the reviews
    - Changed some variable names
    jayjwylie committed Mar 13, 2013
  7. Minor fix for change to AdminClient

    - do not close down AdminStoreClient from queryKeys
    - added some additional checks to test to confirm (non)existence of exceptions&values
    jayjwylie committed Mar 11, 2013
  8. Many minor tweaks to ConsistencyFix code and related files to address…

    … minor review feedback.
    - fixed commenting out of 'protobuff' target
    - add ClientConfig to constructor. This is needed for AdminStoreClient creation. It is confusing that we need both an AdminClientConfig and ClientConfig, but that is because the *ClientConfig code is so clumsy.
    - changed ".stop()" methods to ".close()" to be consistent with other interfaces.
    et cetera
    - Updated all copyright notices that have changed on this branch since December. This touched a ton of files...
    - annotated some TODOs with "(refactor)" to make refactoring todos easier to find.
    jayjwylie committed Mar 11, 2013
  9. Added unit tests for ConsistencyFix, ConsistencyFixWorker, and QueryK…

    Many other fixes and cleanup:
    - tweak many variable names
    - add close method to stop adminClient
    - broke out BadKey to wrap a key with its string representation st failed fixes of badkey's can be dumped in full to file to be retried (without any additional effort)
    - marked 'parseVersion' as deprecated since, if we do this again, we should dump bytes not strings
    - track obsolete version exceptions and various statuses in Stats
    - clean up of arguments, variable names, etc.
    - cleanly close down fixer...
    - more logger.trace output
    - minor cleanup
    - added getVersioned() helper method
    - added consistency-fix store
    - marked all tests as @Test
    - update copyright notice
    jayjwylie committed Mar 8, 2013
  10. Added KeySampler and KeyVersionSampler tools as a first step towards …

    …replacing "entropy" tool. Added another argument to bulk fetch operations that specifies maxRecords so that server can fetch a subset of a partition.
    - Samples keys from a cluster
    - Given file that lists keys per store, samples versions from each "responsible node" for that key
    - passed maxRecords through
    - TODO for future clean up of some types
    - auto generated!
    - white space
    - handle maxRecords
    - handle maxRecords
    - fixed usage of skipRecords
    - added maxRecords
    - added mac_records to protobuff definition
    - added maxRecords field to test
    jayjwylie committed Mar 7, 2013
  11. Made rebalance --show-plan slightly more verbose and added yet anothe…

    …r analysis for cluster balance ("zone primary").
    - print out hostname within plan to make it easier to read (rather than having to lookup node ID)
    - calculate "zone primary" balance to understand which hosted partitions act as pseudo-master when zoned routing is used.
    jayjwylie committed Mar 4, 2013
  12. Review and cleanup of consistency checker.

    - added required argument for an output file name for bad keys
    - changed Reporter to print out 'just the key' to the output file; it
      outputs more info at DEBUG level in general.
    - removed 'quiet' option
    - throw exceptions:
      - if # partitions differ across clusters
      - if replication factor is hinky
      - if isExpired encounters unknown type
    - main catches exceptions and fails fast
    - changed system.out debugging to logger.trace
    jayjwylie committed Feb 26, 2013
  13. Tweaked Rebalancer --output-dir again to better name interim metadata…

    … files for each batch.
    jayjwylie committed Feb 22, 2013
  14. Changed Rebalancer --output-dir option to append numbers to each .xml…

    … file it outputs so that we have access to interim cluster configs.
    jayjwylie committed Feb 22, 2013
  15. Refactored Consistency Check

    zhongjiewu committed with jayjwylie Feb 21, 2013
  16. Default to printing out BADKEYs from ConsistencyCheck. Cleaned up deb…

    …ug/trace messages in
    jayjwylie committed Feb 13, 2013
  17. Fixed hashmap issues in AdminClient raised during code review. Added …

    …'--parse-only' option to ConsistencyFix.
    - Added hashCode & equals methods to AdminClient.Nodestore
    - cleaned up getSocketStore to not leak concurrently created socket stores.
    - added parse only flag which limits that actions of the fixer to bootstrapping and parsing the input file.
    jayjwylie committed Feb 12, 2013
  18. Added 'dry-run' option and cleaned up help message.

    '--dry-run' option goes through all of the read paths (reading files, reading from servers) and calculates what to write where, but does not actually do any writes!
    Should combine --dry-run with these log4j settings:
    jayjwylie committed Feb 11, 2013
  19. Code fixes for the fixing of orphans.

    - added .trace output for parsing of ugly input
    - pass the correct key-type into constructor
    - substantially more .debug output to trace operation
    jayjwylie committed Feb 11, 2013
  20. Actually invoke the BadKeyOrphanReader. Also committing a bunch of

    TODOs for later cleanup.
    jayjwylie committed Feb 11, 2013
  21. Added basic code for repairing orphaned key,values.

    - added BadKeyOrphanReader extends BadKeyReader to consume different
      input file
    - added "orphan-format" flag to indicate that the 'bad-key-file-in' is
      of orphaned key/values.
    - added constructor to take QueryKeyResult of orphaned keys
    - modified resolveReadConflicts to add orphaned key/values to
      imaginary nodes for the sake of determine the value/version to be
    jayjwylie committed Feb 10, 2013
  22. Added per-server throttling to the Consistency Fixer.

    Added a map of EventThrottle objects such that repair traffic to each server can be throttled. We care about throttling write rate because of its potential impact on GC and cleaning.
    jayjwylie committed Feb 9, 2013