Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also compare across forks.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also compare across forks.
base fork: voldemort/voldemort
base: release-080
...
head fork: voldemort/voldemort
compare: release-1.4.2
Checking mergeability… Don’t worry, you can still create the pull request.
This comparison is big! We’re only showing the most recent 250 commits
Commits on Mar 20, 2013
@jayjwylie jayjwylie Fixed minor issue with tests in AdminServiceBasicTest. They were not …
…including an appropriate zone list when constructing a cluster.
10d57ac
@jayjwylie jayjwylie Minor tweak to dealing with contiguous runs of partitions.
- Hard code 10 repeated attempts to get rid of contiguous partitions.
- Added a TODO to do something perfect at some unspecified time in the future...
796aa5d
@jayjwylie jayjwylie Minor fix due to merging of persistency-check and rebalance-bug-fix b…
…ranches
a345a05
@jayjwylie jayjwylie Significant refactoring of util methods. Added helper classes Cluster…
…Instance, StoreInstance, and StoreDefinitionUtil.

ClusterInstance wraps up a Cluster and List<StoreDefinition> object
and provides helper methods that operate on these objects together.

StoreInstance wraps up a Cluster and a StoreDefinition object
and provides helper methods that operate on these objects together.

StoreDefinitionUtil provides helper method that operate on either
StoreDefinition or List<StoreDefinition> objects.

Many methods have migrated out of RebalanceUtils into these new
classes.
7e98b10
@jayjwylie jayjwylie Further refactoring.
src/java/voldemort/utils/ConsistencyFix.java
- rename inner class VoldemortInstance to ConsistencyFixContext
- Drop methods that were duplicated in StoreInstance

src/java/voldemort/utils/StoreInstance.java
- clean up comments for method getNodeIdListForPartitionIdList
6ba10a1
@jayjwylie jayjwylie Added more TODOs from code review feedback. 5db6a1d
@jayjwylie jayjwylie Moved AdminClient.QueryKeyResult into its own file.
src/java/voldemort/client/protocol/admin/QueryKeyResult.java
- A more complete, proper class based on the inner class that had been in AdminClient.java
83b87bc
@jayjwylie jayjwylie Minor comment/todo clean up. 37d3b76
@jayjwylie jayjwylie Fix errors introduced during rebase process.
Rebase with master was ugly. Picked up a bunch of AdminClient refactoring. These changes conflicted with many changes on the persistency-check branch. Had to fix the return type of AdminClient.queryKeys by hand (to iterator<QueryKeyResult>).

Added some TODO comments to AdminClient.HelperOperations methods that should move into ClusterInstance.

Also updated some copyright dates.
79de267
@jayjwylie jayjwylie Added AdminStoreClient helper class to AdminClient.
src/java/voldemort/client/protocol/admin/AdminClient.java
- added AdminStoreClient which manages a ClientRequestExecutorPool and caches pertinent SocketStore (Store<ByteArray, byte[], byte[]>) objects. This allows the admin client to re-use connections when doing basic store operations (put, get, etc) against individual servers.
- added TODOs about other methods I think should be cleaned up
- significantly revised repairEntry
- added a queryKey method that uses the AdminStoreclient

src/java/voldemort/client/protocol/admin/RepairEntryResult.java
- return type for AdminClient.StreamingStoreOperations.repairEntry()

src/java/voldemort/utils/ConsistencyFix.java b/src/java/voldemort/utils/ConsistencyFix.java
- clean up to keep up with changes to AdminClient interfaces & types
- cleaned up TODOs
3c2fe12
@jayjwylie jayjwylie Separate simple admin ops from streaming ops and clean up ConsistencyFix
src/java/voldemort/client/protocol/admin/AdminClient.java
- separated simple StoerOperations from StreamingOperations
- note: renamed 'StreamingStoreOperations storeOps' to 'StreamingOperations streamingOps'
- pulled the exception handling logic out of queryKeys/repairEntries and reanmed to more generic names (getNodeKey and putNodeKeyValue).

src/java/voldemort/client/protocol/admin/QueryKeyResult.java
- made CTors public

src/java/voldemort/utils/ConsistencyFix.java
- significant clean up & refactoring
- match changes in voldemortadmintool

other
- some other TODOS
- other files touched due to renames within AdminClient
02e8063
@jayjwylie jayjwylie Parallelized the consistency fixer. Interim checkin. Need to refactor…
… into separate files.

src/java/voldemort/utils/ConsistencyFix.java
- 1 thread for reading  file of bad keys
- 1 thread for writing any bad keys that are not fixed
- thread pool for sending gets
- thread pool for sending puts
2aea4e6
@jayjwylie jayjwylie Heavily refactored ConsistencyFix. Some key outstanding TODOs remain.
src/java/voldemort/utils/ConsistencyFix.java
- split out much functionality
- switched from having entirely static interfaces to being a normal non-static class

src/java/voldemort/utils/ConsistencyFixCLI.java
- split out CLI aspect from ConsistencyFix

src/java/voldemort/utils/ConsistencyFixKeyGetter.java
- split out thread for getting bad keys from ConsistencyFix

src/java/voldemort/utils/ConsistencyFixRepairPutter.java
- split out thread for repairing bad keys from ConsistencyFix
73d62a1
@jayjwylie jayjwylie Complete implementation of consistency fixer.
src/java/voldemort/utils/ConsistencyFix.java
- added Execute method that orchestrates all the threads
- switched pattern of thread execution:
  - one thread for reading bad keys file that submits to ...
  - a thread pool of workers that enqueues badkeys for the ...
  - one thread writing still bad keys
  - note: construct thread pool with a blocking queue
- switched to logging (rather than System.out/err)
- added Stats tracking
- moved methods that do complicated work (getting & repairing keys) out

src/java/voldemort/utils/ConsistencyFixWorker.java
- moved methods that do complicated work (getting & repairing keys) in

src/java/voldemort/utils/ConsistencyFixCLI.java
- added 'progress-bar' option
- got rid of 'verbose' option
- moved all thread orchestration to ConsistencyFix
44f1737
@jayjwylie jayjwylie Re-commiting a series of ZWu's commits to the consistency fix in a si…
…ngle batch. The other commits were not made against the same master and so this is easier than trying to figure out what went wrong in the merge/rebase.
e5ddb34
@jayjwylie jayjwylie Added per-server throttling to the Consistency Fixer.
Added a map of EventThrottle objects such that repair traffic to each server can be throttled. We care about throttling write rate because of its potential impact on GC and cleaning.
8cb8efa
@jayjwylie jayjwylie Added basic code for repairing orphaned key,values.
src/java/voldemort/utils/ConsistencyFix.java
- added BadKeyOrphanReader extends BadKeyReader to consume different
  input file

src/java/voldemort/utils/ConsistencyFixCLI.java

- added "orphan-format" flag to indicate that the 'bad-key-file-in' is
  of orphaned key/values.

src/java/voldemort/utils/ConsistencyFixWorker.java
- added constructor to take QueryKeyResult of orphaned keys
- modified resolveReadConflicts to add orphaned key/values to
  imaginary nodes for the sake of determine the value/version to be
  repaired
f573b53
@jayjwylie jayjwylie Actually invoke the BadKeyOrphanReader. Also committing a bunch of
TODOs for later cleanup.
1d115da
@jayjwylie jayjwylie Missed afile when doing the hand merge of consistency check stuff. c29bdf5
@jayjwylie jayjwylie Code fixes for the fixing of orphans.
src/java/voldemort/utils/ConsistencyFix.java
- added .trace output for parsing of ugly input
- pass the correct key-type into constructor

src/java/voldemort/utils/ConsistencyFixWorker.java
- substantially more .debug output to trace operation
eceaf01
@jayjwylie jayjwylie Added 'dry-run' option and cleaned up help message.
'--dry-run' option goes through all of the read paths (reading files, reading from servers) and calculates what to write where, but does not actually do any writes!

Should combine --dry-run with these log4j settings:

log4j.logger.voldemort.utils.ConsistencyFix=TRACE
log4j.logger.voldemort.utils.ConsistencyFixWorker=DEBUG
213003a
@jayjwylie jayjwylie Fixed hashmap issues in AdminClient raised during code review. Added …
…'--parse-only' option to ConsistencyFix.

src/java/voldemort/client/protocol/admin/AdminClient.java
- Added hashCode & equals methods to AdminClient.Nodestore
- cleaned up getSocketStore to not leak concurrently created socket stores.

src/java/voldemort/utils/ConsistencyFix(CLI).java
- added parse only flag which limits that actions of the fixer to bootstrapping and parsing the input file.
0c6893c
@jayjwylie jayjwylie Default to printing out BADKEYs from ConsistencyCheck. Cleaned up deb…
…ug/trace messages in ConsistencyFix.java.
104c927
@jayjwylie jayjwylie Added server-put tracking to progress bar. 9d3fb7e
@zhongjiewu zhongjiewu Refactored Consistency Check 433ce02
@jayjwylie jayjwylie Changed Rebalancer --output-dir option to append numbers to each .xml…
… file it outputs so that we have access to interim cluster configs.
1162dbb
@jayjwylie jayjwylie Tweaked Rebalancer --output-dir again to better name interim metadata…
… files for each batch.
b05967b
@jayjwylie jayjwylie Added more info to cluster dump to track which nodes host 'hot' parti…
…tions.
c66a0c2
@jayjwylie jayjwylie Review and cleanup of consistency checker.
- added required argument for an output file name for bad keys
- changed Reporter to print out 'just the key' to the output file; it
  outputs more info at DEBUG level in general.
- removed 'quiet' option
- throw exceptions:
  - if # partitions differ across clusters
  - if replication factor is hinky
  - if isExpired encounters unknown type
- main catches exceptions and fails fast
- changed system.out debugging to logger.trace
cd62a15
@jayjwylie jayjwylie Made rebalance --show-plan slightly more verbose and added yet anothe…
…r analysis for cluster balance ("zone primary").

src/java/voldemort/client/rebalance/RebalancePartitionsInfo.java
- print out hostname within plan to make it easier to read (rather than having to lookup node ID)

src/java/voldemort/utils/ClusterInstance.java
- calculate "zone primary" balance to understand which hosted partitions act as pseudo-master when zoned routing is used.
e50ad0a
@jayjwylie jayjwylie Added KeySampler and KeyVersionSampler tools as a first step towards …
…replacing "entropy" tool. Added another argument to bulk fetch operations that specifies maxRecords so that server can fetch a subset of a partition.

src/java/voldemort/utils/KeySamplerCLI.java
- Samples keys from a cluster

src/java/voldemort/utils/KeyVersionSamplerCLI.java
- Given file that lists keys per store, samples versions from each "responsible node" for that key

src/java/voldemort/client/protocol/admin/AdminClient.java
- passed maxRecords through
- TODO for future clean up of some types

src/java/voldemort/client/protocol/pb/VAdminProto.java
- auto generated!

src/java/voldemort/server/protocol/admin/AdminServiceRequestHandler.java
- white space

src/java/voldemort/server/protocol/admin/FetchStreamRequestHandler.java
src/java/voldemort/server/protocol/admin/FetchEntriesStreamRequestHandler.java
src/java/voldemort/server/protocol/admin/FetchKeysStreamRequestHandler.java
- handle maxRecords

src/java/voldemort/server/protocol/admin/FetchPartitionKeysStreamRequestHandler.java
src/java/voldemort/server/protocol/admin/FetchPartitionEntriesStreamRequestHandler.java
- handle maxRecords
- fixed usage of skipRecords

src/java/voldemort/utils/Entropy.java
- added maxRecords

src/proto/voldemort-admin.proto
- added mac_records to protobuff definition

test/unit/voldemort/client/AdminFetchTest.java
- added maxRecords field to test
7669d86
@jayjwylie jayjwylie Documented correct method of compiling protobuffs by hand. 797eeca
@jayjwylie jayjwylie Actually adding files KeySamplerCLI and KeyVersionSamplerCLI. b0a000d
@jayjwylie jayjwylie Added unit tests for ConsistencyFix, ConsistencyFixWorker, and QueryK…
…eyResult.

Many other fixes and cleanup:

src/java/voldemort/utils/ConsistencyFix.java
- tweak many variable names
- add close method to stop adminClient
- broke out BadKey to wrap a key with its string representation st failed fixes of badkey's can be dumped in full to file to be retried (without any additional effort)
- marked 'parseVersion' as deprecated since, if we do this again, we should dump bytes not strings
- track obsolete version exceptions and various statuses in Stats

src/java/voldemort/utils/ConsistencyFixCLI.java
- clean up of arguments, variable names, etc.
- cleanly close down fixer...

src/java/voldemort/utils/ConsistencyFixWorker.java
- more logger.trace output
- minor cleanup

test/common/voldemort/TestUtils.java
- added getVersioned() helper method

test/common/voldemort/config/stores.xml
- added consistency-fix store

test/unit/voldemort/store/routed/ReadRepairerTest.java
- marked all tests as @Test

test/unit/voldemort/utils/ConsistencyCheckTest.java
- update copyright notice
78e9417
@jayjwylie jayjwylie Many minor tweaks to ConsistencyFix code and related files to address…
… minor review feedback.

build.xml
- fixed commenting out of 'protobuff' target

src/java/voldemort/client/protocol/admin/AdminClient.java
- add ClientConfig to constructor. This is needed for AdminStoreClient creation. It is confusing that we need both an AdminClientConfig and ClientConfig, but that is because the *ClientConfig code is so clumsy.
- changed ".stop()" methods to ".close()" to be consistent with other interfaces.

et cetera
- Updated all copyright notices that have changed on this branch since December. This touched a ton of files...
- annotated some TODOs with "(refactor)" to make refactoring todos easier to find.
ad7ca98
@jayjwylie jayjwylie Minor fix for change to AdminClient
src/java/voldemort/client/protocol/admin/AdminClient.java
- do not close down AdminStoreClient from queryKeys

test/unit/voldemort/client/AdminServiceBasicTest.java
- added some additional checks to test to confirm (non)existence of exceptions&values
96ed494
@jayjwylie jayjwylie Minor fixes for tests that broke due to changes elsewhere in the code. d486f2a
@jayjwylie jayjwylie remove skipRecords from fetching API and protobuf
AFAIK skipRecords was never used. By inspection, the code that would have been exercised if it had been used has never been correct. Removing skipRecords from the code base.

Also:
- Added a number of TODOs to the code from the reviews
- Changed some variable names
a2d9ebb
@jayjwylie jayjwylie change maxRecords to recordsPerPartition in fetch API and protobuf
These are cosmetic changes. The client-side and server-side code does not properly do recordsPerPartition yet.

Added a few TODOs in the code too.
5aca5c0
@jayjwylie jayjwylie Correctness fixes and significant refactoring of Fetch*StreamRequestH…
…andlers. Expanded AdminFetchTest.

Added more common helper methods to common base class of all fetchers FetchStreamRequestHandler.

Added abstract base classes for partition-based fetching and non-partition-based fetching:
- FetchPartitionStreamRequestHandler (partition-based)
- FetchItemsStreamRequestHandler (non-partition-based)

Refactored some code up to abstract base classes and made implementations as similar as possible (without heroic efforts) across all fetchers:
- FetchEntriesStreamRequestHandler
- FetchKeysStreamRequestHandler
- FetchPartitionEntriesStreamRequestHandler
- FetchPartitionKeysStreamRequestHandler

Significant better test coverage in AdminFetchTest
- tests fetching keys as well as fetching entries
- tests partition-aware and non-partition-aware servers
- tests per-partition limits on entries/keys fetched

All of this clean up and additional testing lead to minor correctness fixes.

Minor other clean ups of comments, override annotations, and fixes for KeySamplerCLI.
96d1513
@jayjwylie jayjwylie Addressed all review feedback and TODOs for KeySamplerCLI
KeySamplerCLI
- added options: --store-names, --partition-ids, --keys-per-second-limit, and --progress-period-ops
- got rid of unnecessary (and weird) retry loop. Can add seomthing like that later if needed.
- pass all partitions to fetcher now instead of one-at-a-time

Also did cosmetic fixes for KeyVersionSamplerCLI and Entropy.java
f5e8f5a
@jayjwylie jayjwylie Addressed review feedback and TODOs for KeyVersionSamplerCLI (and ren…
…amed it to KeyVersionFetcherCLI).

- mostly usability changes about command line options...
- one copyright fix
92b80c0
@jayjwylie jayjwylie Interim commit to rename a bunch of files. 96862f2
@jayjwylie jayjwylie Addressed all code review comments for KeySampler and KeyVersionFetch…
…er. Renamed many classes and methods related to FetchStreamRequestHandler.

- All sub-classes of FetchStreamRequestHandler have been renamed to have a more consistent nomenclature.
- Did some further refactoring in the FullScan* classes to move more work from leaf classes to FullScanFetchRequestHandler.java
- moved scan accounting to overall bae class
- Added getNodesPartitionIdForKey method to StoreInstance to help with some fetch logic
1a0b8f7
@jayjwylie jayjwylie TODOs for refactoring and copyright header updates. 9c75277
@jayjwylie jayjwylie Added pending release notes for this merge to master. d0023e9
Commits on Mar 22, 2013
@jayjwylie jayjwylie Fix to log4j set up in bin/run-class.sh that should correctly configu…
…re log4j.
667ed50
@jayjwylie jayjwylie Reduce chatter in logs while a node is unavailable (from INFO to DEBU…
…G level output).
8f4e8b0
Chinmay Soman Using a per file checksum generator in the file copy in HdfsFetcher. …
…This is used to handle the case where we might retry the copy in case of a Filesystem (hdfs) error.
b8f3a8d
Chinmay Soman Adding an extra catch block for Exception and Throwable types. This i…
…s used to catch the ClassNotFound exceptions
1c606ef
@abh1nay abh1nay Adding test cases whch simulate intermittent exceptions etc
in the hdfsfetcher and simulate retry logic
this ensures checksum calculation is robust
25d6298
@abh1nay abh1nay Added test case to test runtime exceptions
Cleaned up the try catch  logic
11dff85
Commits on Mar 25, 2013
@abh1nay abh1nay Minor changes to test case# Please enter the commit message for your …
…changes. Lines starting
d4f5935
@abh1nay abh1nay last commit did not pick up the actual fix for the test case-- checki…
…ng ti in
0c05a03
@abh1nay abh1nay Updated release notes e0e4f9b
@abh1nay abh1nay Updated release number 3736b4a
@abh1nay abh1nay fixed run-class.sh to take absolute paths 314e2e8
Commits on Mar 26, 2013
Chinmay Soman Basic working prototype for Coordinator and Sample Thin client 4140295
Chinmay Soman Added the coordinator package. Modified Benchmark to use thin client 939d2cb
Chinmay Soman Creating FatClientWrapper for Thread pool isolation. 4cd607d
Chinmay Soman First working version of Coordinator service. Includes REST request a…
…nd response handling, Error handling, automatic checking of metadata changes, fat client config management
572f87b
Chinmay Soman Fixed .classpath which had an illegal entry 791ae59
Chinmay Soman A working implementation of the Coordinator and thin client. Includes…
… following things:

- Creating AbstractStore and AbstractStorageEngine to refactor
  the corresponding Store and StorageEngine interfaces.
- Refactored the fat client to accomodate dynamic per call timeout.
- Isolated Fat client wrapper to safeguard multitenancy
- Autobootstrap mechanism added to the Coordinator service
- Basic HTTP request/response parsing and Error handling
9785e68
Chinmay Soman Adding the missing R2Store file e7c2ae9
Chinmay Soman Moving thin client to contrib. Also fixing Benchmark to use DefaultSt…
…oreClient instead of Thin client
d0f91f6
Chinmay Soman - Added GetAll and Delete implementations on the Coordinator (and the…
… temporary rest client)

- Converted Coordinator into an AbstractService and added CoordinatorConfig
- Refactored Composite Voldemort request into different types
4964e7d
Commits on Mar 31, 2013
@ctasada ctasada Merge remote-tracking branch 'voldemort/master' 90c0b5f
Commits on Apr 03, 2013
@zhongjiewu zhongjiewu make run-class.sh working in Mac per ctasada ceb7de8
Commits on Apr 04, 2013
Chinmay Soman Added unit tests to ensure that slops are registered for different as…
…ynchronous put operation failures
e9f0776
Commits on Apr 05, 2013
Chinmay Soman Bug fixes to HintedHandoffFailureTest and added more tests to handle …
…3-2-2 config. Removed SleepyForceFailStore
7aa96cc
Commits on Apr 10, 2013
@vinothchandar vinothchandar Tool to forklift data over for store migrations 7234b65
Commits on Apr 11, 2013
@vinothchandar vinothchandar Adding unit test for fork lift tool 560bd67
@vinothchandar vinothchandar Clarifying arbitrary choice to return BEFORE for equal vector clocks. 74bdfde
Commits on Apr 12, 2013
@vinothchandar vinothchandar Forklift tool fix to equally spread fetches b8e6f78
Commits on Apr 15, 2013
Chinmay Soman Adding another Hinted handoff failure test to ensure main thread retu…
…rns with failure when all replicas dont respond
18c2931
Chinmay Soman Added bigger timeout to testNoSlopsOnAllReplicaFailures test and doin…
…g better exception handling in HintedHandoffFailureTest
4b1557b
Chinmay Soman Added Null pointer check in teardown of ReadOnlyStorageEngineTest db6ac44
Commits on Apr 17, 2013
@ctasada ctasada Merge commit 'db6ac447895255c84ef0e0cbd0303ffa6b45e05b' 266221c
Commits on Apr 18, 2013
@vinothchandar vinothchandar Adding basic support for abortable rebalances, more to follow fb8f1a8
@vinothchandar vinothchandar Adding proxy write tests 927b02c
@vinothchandar vinothchandar Proxy to donor implementation + tests d24d0b1
@vinothchandar vinothchandar Fixing test cases, have RebalanceController use ProtoBuf 881d7b3
Commits on Apr 22, 2013
@vinothchandar vinothchandar Adding @deprecated for proxy put param + cosmetic debug msg fixes 736ccc8
Commits on Apr 23, 2013
@abh1nay abh1nay Added a mode to the forklift tool to allow all conflicting versions
to be copied over to the destination
800d48f
@abh1nay abh1nay Added unit test b245e3b
@abh1nay abh1nay Cleaned up code# Please enter the commit message for your changes. Li…
…nes starting
ff2d58f
Chinmay Soman Making the methods of RebalancePartitionsInfo synchronized so as to a…
…void the race condition in asMap and removeStore
77e08d7
Chinmay Soman Removing synchronized keyword from the static and private methods cc89784
Commits on Apr 24, 2013
@vinothchandar vinothchandar Zone N ary helpers + StoreRoutingPlan tests 626ea52
Chinmay Soman Adding a missing else block for throwing a store def mismatch exception 9991605
@abh1nay abh1nay Remove redundant callbacks 29da9ef
@abh1nay abh1nay Cleaned up logger statements aefddee
@abh1nay abh1nay More cleanup# with '#' will be ignored, and an empty message aborts t…
…he commit.
6b5f854
Commits on Apr 25, 2013
@abh1nay abh1nay Updated release notes 6ca2f83
Commits on Apr 26, 2013
Chinmay Soman * Added monitoring (JMX) for fat client wrapper, netty server and req…
…uest handler

* Fixed some bugs and cleanup
* Added unit tests for dynamic timeout store and REST API validation
e6e9e7e
Chinmay Soman Moved slow store to single-slow-store.xml. Previously it was breaking…
… the existing tests
06c4b3e
Chinmay Soman Added the ability to do Versioned puts. Added another test case for t…
…he same
4f097fe
Chinmay Soman Standardized config and cleaned up some code 18e3346
Commits on May 01, 2013
Chinmay Soman Fixing the faulty read-repair logic (removed duplicates, removed unne…
…cessary repairs). Added a test case to verify this in the presence of concurrent versions
be4dbc0
Commits on May 03, 2013
@vinothchandar vinothchandar Added hook to checkpoint all BDB envs c5a143f
@vinothchandar vinothchandar Fix Typo 73802e0
Commits on May 08, 2013
@vinothchandar vinothchandar reimplement proxy put based on zone n-ary replicas logic 82f97c4
@vinothchandar vinothchandar Eliminate unnecessary proxy fetches 63beb58
@vinothchandar vinothchandar Follow fixes to abortable rebalancing f31e48f
@zhongjiewu zhongjiewu vector clock fix 8fd7ea6
@zhongjiewu zhongjiewu using treemap to store clocks in VectorClock 3a5e6f6
Commits on May 13, 2013
@vinothchandar vinothchandar Implementing atomic multi version puts to storage b743678
@vinothchandar vinothchandar Rewrite of InMemoryStorageEngine + config to control multiVersionPuts b3147d3
@vinothchandar vinothchandar 1.Enabling proxy puts by dafault
2. Bug fix in proxy put stats
3. Changing order of state change updates for correctness
4. Setting proxy put tests to do one batch rebalancing
e7ecec1
Commits on May 15, 2013
@vinothchandar vinothchandar Rewording comments 22686c8
Commits on May 16, 2013
@zhongjiewu zhongjiewu added additional logging; minor change in test store classes 7e60cc9
@zhongjiewu zhongjiewu added new hintedHandoffSendHint test and remove old hintedHandOff test cbd4dbd
@zhongjiewu zhongjiewu refactored PerformParallelPutAction and related to ensure a slop is r…
…egistered
1ba83a1
@zhongjiewu zhongjiewu more refactoring to make sure response is handled once and only once 208e78c
@zhongjiewu zhongjiewu added end-to-end test for slops a1ac2dd
@zhongjiewu zhongjiewu additional logging 7ef135d
@zhongjiewu zhongjiewu additional slop fix c1400bd
@zhongjiewu zhongjiewu deprecate send hint serial 204615f
@zhongjiewu zhongjiewu more commits on slop fix ffe892d
Commits on May 17, 2013
@abh1nay abh1nay Allow update metadata to take both stores and cluster xml 6250c10
@abh1nay abh1nay atomic update of stores and cluster xml during rebalance 818d96b
@abh1nay abh1nay Added new end to end test for verifying the atomic update is consistent
on bootstrap
cleaned up code based off last code review
11ea2f5
@abh1nay abh1nay Made the getserverStateLocked explicit
adding the new test case this time around
34ebf5a
@abh1nay abh1nay Cleaned up the test 361bc02
@abh1nay abh1nay more cleanup on the test aaf95d0
@abh1nay abh1nay cleanup on tests 995b30a
@abh1nay abh1nay Fix unused variable fbe2589
Chinmay Soman Adding a null check for the versioned value object in convertStringTo…
…Object in MetadataStore. This was causing a small problem while restarting the Voldemort server
d91c249
Commits on May 20, 2013
@voldemort Make sure elapsed time is not negative
System.nanoTime() can sometimes go backwards as it relies on
performance counters. This commit fixes exceptions that can
surface due to requestTime being negative. See github issue:
#139
81f3975
Commits on May 21, 2013
Chinmay Soman Small bug fixes and cleanup in R2Store fc89e8b
@voldemort Add MANIFEST.MF to git ignore
Manifest.mf is an autogenerated file. Adding it to the ignore list.
a5ba641
Commits on May 24, 2013
@vinothchandar vinothchandar Fixes to KeySampler/KeyVersionFetcher
-- Handle larger fetches without OOME
-- Adding support to use query-keys on hexstring keys, so as to be compatible with the keySampler/keyVersionFetcher
093d89c
Commits on May 29, 2013
@jayjwylie jayjwylie Dropped META-INF/MANIFEST.MF a53be08
@voldemort Upgrade Google Collections lib to Guava lib 797b7c6
Commits on May 30, 2013
@ctasada ctasada Merge branch 'master' of https://github.com/voldemort/voldemort
# By Vinoth Chandar (16) and others
# Via Siddharth Singh
* 'master' of https://github.com/voldemort/voldemort: (56 commits)
  Upgrade Google Collections lib to Guava lib
  Dropped META-INF/MANIFEST.MF
  Fixes to KeySampler/KeyVersionFetcher
  Add MANIFEST.MF to git ignore
  Small bug fixes and cleanup in R2Store
  Make sure elapsed time is not negative
  Adding a null check for the versioned value object in convertStringToObject in MetadataStore. This was causing a small problem while restarting the Voldemort server
  Fix unused variable
  cleanup on tests
  more cleanup on the test
  Cleaned up the test
  Made the getserverStateLocked explicit adding the new test case this time around
  Added new end to end test for verifying the atomic update is consistent on bootstrap cleaned up code based off last code review
  atomic update of stores and cluster xml during rebalance
  Allow update metadata to take both stores and cluster xml
  more commits on slop fix
  deprecate send hint serial
  additional slop fix
  additional logging
  added end-to-end test for slops
  ...
97289e6
@ctasada ctasada Solves compilation problem with Java 7 113d234
Commits on Jun 03, 2013
@zhongjiewu zhongjiewu Merge pull request #146 from ctasada/issue132
Issue 132 - Java 7 Compilation Bug tested for compilation and smoke test
8efee8a
Commits on Jun 04, 2013
@vinothchandar vinothchandar Adding a Production worthy sample configuration
do a
   $ bin/voldemort-prod-server.sh config/prod_single_node_cluster
to get started
ea7cc0c
Commits on Jun 05, 2013
@abh1nay abh1nay Reverting back change to on wire protocol for update metadata
to preseve backwards compatibility
1dc6872
Commits on Jun 06, 2013
@voldemort Check that zone value for a node is not null 96ff6d9
@abh1nay abh1nay Added functionality to get store schema
Example:
==========

 curl http://localhost:8080/schemata/dGVzdA==
{"key-serializer": "SerializerDefinition(name = string, schema-info =
{}, compression = null)", "value-serializer": "SerializerDefinition(name
= string, schema-info = {}, compression = null)"}
b8af33d
@abh1nay abh1nay Cleaned up code based on Vinoth's review f6e7b9c
@vinothchandar vinothchandar Adding sample configs for Voldemort Coordinator 8db2a31
Commits on Jun 11, 2013
@vinothchandar vinothchandar Implement metadata bootstrapping for Java Rest Client 3f04f58
Commits on Jun 14, 2013
Chinmay Soman Bug fix in RESTErrorHandler - adding the right content length in the …
…response header
74d71c9
Commits on Jun 15, 2013
@vinothchandar vinothchandar Update CONTRIBUTORS
Long missing update for the current Voldemort Team at Linkedin
d90118f
Commits on Jun 18, 2013
@abh1nay abh1nay Fixed case with statsTrackingStore not being wrapped around properly
with DynamicTimeoutStoreClient since ser. and compression were disabled
f23d03b
@abh1nay abh1nay For some reason the method that I had added to retrieve storedefs was
missing adding it back
f84b947
@abh1nay abh1nay Some cleanup 4846ecc
@abh1nay abh1nay Added comments b8a8a4f
Commits on Jun 20, 2013
@voldemort Add more locations to gitignore 7daac20
@voldemort Remove unneeded check
adminClient doeesn't change in the try block so no
check needed in the finally block.
c809fba
Chinmay Soman Added 3 zone tests to RoutedStoreTest.java, fixed a small bug in GetA…
…llConfigureNodes
5f796e8
Chinmay Soman Removed BannagePeriodFailureDetector parameter from RoutedStoreTest a…
…nd some beautification changes
ded4506
@jayjwylie jayjwylie Partition balance analysis tool for more than two zones.
- Started a directory for all of our tools (src/java/voldemort/tools)

- PartitionBalance :
  - broke this util out of ClusterInstance to make it more self-contained
  - this changed the declaration of a lot of the methods that invoked analyzeBalance and analyzeBalanceVerbose

- ClusterInstanceTest  :  Added a bunch of tests to confirm that partition balance can be constructed (or not) for combinations of 2 & 3 zones
34e7a4b
@jayjwylie jayjwylie Added RepartitionCLI. It is the subset of the RebalanceCLI that does …
…repartitioning.
4f1c00b
@jayjwylie jayjwylie Initial version of repartitioning tool 'RepartitionCLI'.
RebalanceCLI
- removed all "generate" options and code from this tool (now tools/RepartitionCLI)
- removed "analysis" option from this tool (now tools/PartitionAnalysisCLI)
- Fixed errors in help/usage docs

RepartitionCLI
- Removed unnecessary options that restricted partition movement to be 'within a zone'. This decouples the repartition tool from the planning tool. This also assumes that optimizations can be done at plan time to keep all movement within a zone (when possible).
- cleaned up option names to be more concise
- Added options for target cluster/stores to allow zone expansion to be fully specified

PartitionBalance
- New class extracted from ClusterInstance
- Made all partition analysis zone-aware. This prevents zones with fewer nodes, or partitions, from skewing any analysis. I.e., each zone's balance analysis is now correctly normalized to be comparable with other zones' balance analyses.
- added inner ZoneBalanceStats class that makes stats tracking clearer
- added getUtility methods to hide the exact utiliyt method from the interfaces
- Changed utility methods to sum over all zones and to combine zone primary analysis with nary analysis.

RebalanceClusterUtils.java
- changed/extended balancePrimaryPartitions to handle zone expansion
- Stripped out a bunch of unnecessary options/code. I.e., options/code that tried to minimize cross zone moves during repartitioning (rather than during planning!)
- Added some helper methods to clean up repeated code.

RebalanceUtilsTest
- test new helper methods
92c1139
@jayjwylie jayjwylie Refactoring. Renamed RebalanceClusterUtils->RepartitionUtils and move…
…d some tests into new file RepartitionUtilsTest.
ce5a73e
@jayjwylie jayjwylie Added tests of repartitioning algorithms.
RepartitionCLI
- added arg for greedy zone IDs; not bothering exposing it via CLI though

ClusterUtils.java
- split method that identifies contiguous partition ID runs into two, better tested and now correct, methods

PartitionBalance
- fix div by zero error

RepartitionUtils
- Documented the repartition method
- Reused the internal balancePrimaryPartitions method in the method that breaks apart contiguous partitoin runs
- dropped unused methods
- switched greedy algorithms to do either zone-by-zone or cluster as a whole. unless we find cluster-as-a-whole leads to too much data movement, is the behavior that we want.

ClusterInstanceTest
- made all the get*Cluster and get*StoreDefs methods static and added some more. These are generally useful for tests beyond ClusterInstance.

RepartitionUtilsTest
- Actually test all of hte repartitioning algorithms!
722d9af
@jayjwylie jayjwylie copyright fix 0e17e99
@jayjwylie jayjwylie Initial work on RebalancePlanCLI.
Extracted the planning logic out of RebalanceController. This will reduce cut-and-paste coding, clarify which arguments are needed to tune planning (versus execution of a plan), and isolate rebalance planning logic from execution logic. There are a ton of TODOs added in this commit. As the refactoring continues, and the controller is changed to use the new RebalancePlan, these TODOs will be addressed. This commit has a working RebalancePlanCLI that handles the use cases rebalance planning has historically handled (rebalance in place and cluster expansion). This commit does not address zone expanison. The output of plan statistics is much richer and includes detailed statistics on cross zone moves (from which zone to which zone) and nodes (how much in and how much out). All these statistics are at the partition-store level which should be more informative than prior statistics. Finally, these statistics include an estimation of storage overhead per-node which can be used to assess how much free disk space is required to execute a plan safely.

OrderedClusterTransition
- TODOs to deprecate this. This logic belongs in the plan, not the execution.

RebalanceClusterPlan
- TODOs for refactoring
- Has logic for doing things "the old way" and the "new way". The new way has simpler data structures nad better encapsulation.
- Many more getters for detailed plan stats

RebalanceNodePlan
- Minor TODOs and some renaming of varaibles

RebalancePartitionsInfo
- A ton of refactoring TODOs
- Fixed all move counting getters to count partition-stores

Node
- getStateString method to pretty print ID, host name and ports.

RepartitionCLI
- more argument handling

PartitionBalance
- getters for partition-store accounting of current cluster+store defs

RebalanceUtils
- Some methods to deprecate
- validation methods for cluster arguments

RebalancePlan
- New class to encapsulate all rebalancing planning

RebalanceTypedBatchPlan
RebalanceDonorBasedBatchPlan
RebalanceStealerBasedBatchPlan
- Sub-classes of RebalancePlan that are needed for converting static plan into executable plan

RebalancePlanCLI
- CLI for generating a plan

MoveMap & MoveMapTest
- 2-d map counter used in partitoin-store accounting
aee0639
@jayjwylie jayjwylie I mangled the merge after a rebase. This commit manually fixes the er…
…rors I introduced. Mea culpa.
857aa57
@jayjwylie jayjwylie RebalancePlanCLI working for 2-to-3 zone expansion
Stealers steal from first donor with desired partition-store. This currently biases all work towards donors with lowest node IDs. This is truly imbalanced plan!

RebalanceClusterPlan
- Decorated with more TODOs...
- Changed stealer to pick first donor with desired partition-store
- Added commented-out code section that outlines next step for better planning

RebalancePartitionsInfo
- Decorated with more TODOs

RebalancePlan
- Much clean up refactoring. Construction now generates plan.
- Methods added to get the plan (getPlan) and to print the plan (toString)

RebalanceUtils
- Helper method that derives targetCluster from initialCluster and finalCluster.
5e72e52
@jayjwylie jayjwylie RebalancePlan de-insanification
This is a messy commit:

1) There are a ton of TODOs, a ton of code marked to be deprecated, and a lot of gymnastics to continue using data structures that ought to be simplified or removed. All of this is necessary until the RebalanceController is switched to use the RebalancePlan, and until the abortable rebalance work and this work are merged to master. Cleaning up shared data structures is too high risk while on separate branches.

2) Some of the RebalanceTest tests are currently failing. Going to debug and fix in next commit.

Commit notes:

RebalanceClusterPlan
- renamed cluster members to be target & final rather than current & target
- determines batch plan at construction time
- added batchPlan() method that
  - is more clear than prior planning method
  - optimized to do n-ary to n-ary migration within a cluster so that only new zones require cross zone moves

RebalancePlan
- minor change for method name change

StoreRoutingPlan
- added internal member & method to track all nary-partition-stores per node
- get zone replica type methods that take partition id as input

RebalanceUtils
- added / refactored suite of methods that validate pairs of clusters (current&target, current&final, target&final)

There are tests missing at this point in development:
- simple & basic tests of RebalancePlan
- test StoreRoutingPlan.getNaryPartitionIds (and possibly other methods too)
- tests for RebalanceUtils.validate.* methods
03453dc
@jayjwylie jayjwylie Fixed broken junit tests. Documented and commented out other parts of…
… unit tests that rely on donor-based rebalancing or that rely on partition-stores being deleted during rebalance. Both features are currently (necessarily) broken.

AdminClient
- Fix latent bug that treated node ids as dense/contiguous

RebalanceclusterPlan
- minor refactor

RebalanceController
- handle the case of an "empty" plan differently. Because unneccessary moves are optimized out at plan time, some of the small tests now yield plans with no data movement. I.e., primary partition IDs move among nodes, but data does not because all nodes already host all data. This exposed a corner case in which rebalance was successful, but servers never updated their cluster xml with new partitoin ID mapping.
- Removed random/unnecessary 10s sleep from control path. Annotated with a TODO for discussion to confirm this is right thing to do.

DonorBasedRebalanceAsyncOperation
- minor code clarification

DonorBasedRebalancePusherSlave
- Fixed latent bug (wrong value passed to sleep). Added TODO wondering why this sleep is here in the first place.

StorageEngine
- How are isPartitoinScanSupported() and isPartitionAware() related?

PartitionPrefixedBdbStorageEngine
- add isPartitionAware that returns true. Is this correct!?

AdminRebalanceTest
- Commented out asserts that test deletion during rebalancing

ZonedRebalanceTest
- Removed donor-based rebalancing tests from this test suite.
622ee42
@jayjwylie jayjwylie Added unit tests for RebalancePlan and for cluster xform & verificati…
…on methods in RebalanceUtils

RebalancePlanTest
- Test the core use cases for rebalance planning:
  - no-op,
  - rebalance (zone/node topology same, but partition id layout changed),
  - cluster expansion (zone topology same, but new nodes added & partitions moved to them),
  - zone expansion (zone topoloigy changes with partitions moved to new zone).

RebalanceController
- rename method

RebalancePlan
- added getters for aggregate plan statistics

ServerTestUtils
- Changed getLocalZonedCluster to accept array of node IDs. Necessary to generalize other test methods.

ClusterInstanceTest
- pass storage type into helper methods that construct store defs
- chnagned all get???Cluster.*() methods to use new getLocalZonedCluster interface. Also added more such helper methods.

RebalanceUtilsTest
- Test the cluster transformation & verification methods

RepartitionUtilsTest
- simple clean up
4f0087f
@jayjwylie jayjwylie copyright updates. 80dc13a
@jayjwylie jayjwylie Addressing reviews of balance analysis & repartitionerCLI
RebalancePlan
- print out invalid metadata rate analysis

Cluster
- added members that allow reverse lookup of partitionID to either ZoneId or NodeId

StoreRoutingPlan
- Added reverse lookup of nodeID to Zone Primaries hosted.

RepartitionCLI
- moved constants into RepartitionUtils
- Added validation of clusters & storedefs passed in.

ClusterUtils
- use lookup from Cluster object
- changed all contig partition ID methods to call the one method that does all the complicated work.

PartitionBalance
- Added constants for use in the utility function
- Broke the core, complicated method into small, reasonably-sized pieces.
- Added verbose print out of ZoneReplicaTypes (to complement old "replica type" print out)

RepartitionUtils
- Document the recommend default constants for repartitioning
- Dump invalid metadata rate
- Clean up comments and java doc
- dumpInvalidMetadataRate analyzes and pretty-prints how many pseudo-masters per-zone per-unique-store-def lead to invalid metadata exception.
fdc0eb3
@jayjwylie jayjwylie Refactoring: cleaned up util methods and moved them to appropriate ut…
…il classes

RebalanceUtils
- all dump cluster methods are now here with more consistent interfaces
- analyzeInvalidMetadataRate is here (for lack of better place to put it)

RepartitionUtils
- moved all helper/util methods out of this class

Utils
- added removeItemsToSplitListEvenly, distributeEvenlyIntoList, and distributeEvenlyIntoMap

Moved test methods around to match new locations of util methods.
57686e4
@jayjwylie jayjwylie Additional minor refactoring of utils and utils tests. cb5d292
@jayjwylie jayjwylie Refactoring: Renamed some classes and reorganized their location
RepartitionCLI -> RepartitionerCLI
RepartitionUtils -> Repartitioner

PartitionBalance & Repartitioner : voldemort/utils -> voldemort/tools

Test classes & files updated to correspond with src/java.
54e48c5
@jayjwylie jayjwylie Refactoring: More class renaming and file reorganization
ClusterInstance has been dropped.

ClusterInstanceTest is now PartitionBalanceTest and is in tools.

ClusterTestUtils is a new util class. All of the getFOOClusterBar and getFOOStoreDef(s)BAR methods have been moved into this test helper.
8d5d076
@jayjwylie jayjwylie Minor changes inr esponse to review feedback and documenting a set of…
… tests that are known to fail at this time.
0515ae4
@jayjwylie jayjwylie Initial commit of new rebalance controller.
This is an interim commit. The new RebalanceControllerCLI has been added. Portions of the RebalanceController have been updated to use the RebalancePlan. All the tests that should pass, still pass for the old RebalanceController. About to switch tests to use new RebalanceController that works with a RebalancePlan.
464df36
@jayjwylie jayjwylie Unit tests work with new rebalance controller.
RebalanceController
- helper method to construct RebalancePlan based on cluster read from servers
- divide by zero fix
- many TODOs and Deprecate annotations...

.*RebalanceTask
- drop RebalanceclientConfig from construction and direclty pass in the config parameters needed.

*Rebalance*Test
- do not test donor-based rebalancing. (TODOs to add it back.)
- do not test that keys are actually deleted when delete is true in rebalancing.
- use the new controller/plan code path in all rebalancing unit tests.
79ca1e0
@jayjwylie jayjwylie Switched to new RebalanceController based on RebalancePlan and droppe…
…d tons of now deprecated code.

Notes:
- delete of partition-stores during rebalancing has been removed and is no longer an option.
- the old RebalanceCLI now only offers the entropy tool features. cannot be dropped until SREs are OK with KeySampler/KeyVersionFetcher tool chain for verifying rebalances.
- Dropped RebalanceClientConfig. Configuration parameters are now explicit in pertinent constructors.
- Tests are failing or disabled:
  - donor-based rebalancing is disabled in test suites. Does not currently work with new rebalancing.
  - RebalanceClusterPlanTest tests mostly fail at this time.
  - RebalanceTest.serverSideRouting[1] may timeout
  - ZonedRebalanceTest.testProxyGetDuringRebalancing[1] may timeout

OrderedClusterTransition
- many small changes to work with all other changes
- this class is to-be-deprecated regardless

RebalanceClusterPlan
- removed tons of deprecated code that permitted old controller to continue to work

RebalanceController
- Now uses RebalancePlan
- No longer offers delete-after-rebalancing feature
- Dropped tons of deprecated code...

RebalanceTypedBatchPlan
RebalanceDonorBasedBatchPlan
RebalanceStealerBasedBatchPlan
- no longer extends RebalanceClusterPlan
- the classes that make a generic plan executable as either stealer- or donor-based
- still need to be renamed so purpose of these key classes is more clear

RebalancePlanCLI
RebalanceControllerCLI
- moderate clean up

RebalanceUtils
- dropped unnecessary helper mehods

*Test
- Signficant clean up:
  - use Rebalance*BatchPlan and RebalancePlan
  - don't check keys deleted since no longer supported
70f8259
@jayjwylie jayjwylie Continued refactoring and clearing out deprecated code.
OrderedClusterTransition
- removed unnecessary members:
  id
  currentCluster
  targetCluster
  rebalanceclusterPlan
  storeDefs

RebalanceController
- renamed some member variables for sake of clarity
- switched from orderedClusterTransition.getId to batchCount to uniquely ID batches of work...

RebalanceNodePlan
- added TODOs
- removed unnecessary members:
  attempt
  storeToReplicaToDeletePartitionList

Patched up all tests to not have 'compile time' errors. Have not run tests though...
c1c367e
@jayjwylie jayjwylie interim commit. Some work on taking logic of OrderedClusterTransition…
… and moving it into ExecutableBatches. Logic is broken though, and, more importantly, unnecessary. Given proxy-gets server from local zone and there can only be nominal perf gain from ordering rebalancing tasks at node level, OrderedClusterTransition needs to be removed from teh code, not incorporated into new planning/execution code.
d9e2623
@jayjwylie jayjwylie Interim commit: OrderedClusterTransition has been removed from the co…
…de base.
91be6db
@jayjwylie jayjwylie Interim commit before dropping ExecutableBatch objects
RebalanceClusterPlan
- removed TODOs about trying to "intelligently" order rebalance tasks

RebalanceController
- switched to just using REbalancePlan until tasks are actually constructed that are either stealer- or donor-based.

RebalancePartitionsInfo
- static method for pretty printing a list thereof

*Test
- clean up to deal with other type & interface changes...
d6cd930
@jayjwylie jayjwylie Dropped Rebalance*BatchPlan.java, the classes I thought we needed to …
…encapsulate an executable plan.
42d3868
@jayjwylie jayjwylie Dropping RebalanceNodePlan, the class that sorted lists of RebalanceP…
…artitionsInfo by node id (either stealer or donor node), and then flattened the sorted list out and returned it again. AKA the identity operation on a list of RebalancePartitionsInfo.
2b69609
@jayjwylie jayjwylie Fix on-wire protocol for RebalancePartitionInfoMap
Completed deprecation of member 'attempt' of RebalancePArtitionInfoMap by making it optional and prefixing name with OBSOLETE.

This leads to big changes to auto generated file VAdminProto.java.

Some other minor tweaks to RebalancePArtitionsInfo and comments as well.
083a3e8
@jayjwylie jayjwylie Fixed RebalanceClusterPlanTest to be junit 4 and to work as well as p…
…ossible at this time. Test needs to be re-written to test new planning rather than old planning.
1219576
@jayjwylie jayjwylie Renamed RebalanceClusterPlan(Test).java to RebalanceBatchPlan(Test).j…
…ava. Cleaned up TODOs and comments and variable names in RebalanceBatchPlan.
8121c78
@jayjwylie jayjwylie Cleaned up existing TODOs and added some javadoc comments. a2b950c
@jayjwylie jayjwylie Renamed Rebalance(Long)Test to NonZonedRebalanceLongTest.java. Added …
…ZonedRebalanceLongTest.java.
039b7fc
@jayjwylie jayjwylie Clean up Rebalance(Plan|Controller)CLI
RebalancePlan(CLI)
- set batch size default to INTEGER.MAX_VALUE

RebalanceControllerCLI
- Cleaned up handling of all optional arguments
8a9a791
@jayjwylie jayjwylie Remove no-longer needed test from RebalancePartitionsInfoTest; annota…
…ted another test as mostly failing..
4fdeaf2
@jayjwylie jayjwylie Address review comments for RebalancePlan
RebalanceBatchPlan
- factored logic to decide which donor to steal from out of constructBatchPlan.
- huge header comment about other policies to consider

RebalanceController
- dropped comment about historic (arbitrary) sleep command

RebalancePlan
- added helper hacky method to "clone" cluster

RebalanceCluster
- TODO to add a .clone() method

StoreRoutingPlan
- cleaned up javadoc, TODOs, and method names

DonorBasedRebalancePusherSlave
- set sleep to 30 seconds. Made comments and logger messages consistent with code.

StorageEngine
- cleaned up javadoc for isPartitionAware()

RebalanceUtils
- fix validation method to use safe(r) comparison.

RebalanceBatchPlanTest
- fixed tests to (mostly) only test plan invariants. Prior test code focussed on exact plan details and so was hard-coded to the implementation, rather than the interface.
5244216
@jayjwylie jayjwylie Minor changes to address second round of review feedback on rebalance…
… plan.
8bc5e17
@jayjwylie jayjwylie Fix to store routing plan to correctly track zone-primaries per-node. ad4b726
@jayjwylie jayjwylie Addressed review feedback on new RebalanceController. 91c0b18
@jayjwylie jayjwylie Added unit tests for RebalancePlan, RebalanceBatchPlan, and AbstractZ…
…onedRebalanceTest.

Added following rebalance test case coverage:
- RebalancePlanTest / RebalanceBatchPlanTest / AbstractZonedRebalanceTest
- no-op / shuffle / cluster expansion / zone expansion
- 2-zone / 3-zone

Also did some clean up of rebalance test utils and rebalance tests in general.
c228e5f
@jayjwylie jayjwylie Fixes after rebase/dropped propagateCluster method
After the rebase with the atomic update of cluster/stores branch, had
some clean up to do. Got code running and unit tests passing again. Note
that VAdminProto.java had to be regenerated after the merge.

Noted that RebalanceUtils.propagate cluster was unnecessary (and
potentially dangerous) and so dropped it from the code.
8bbf529
@jayjwylie jayjwylie RebalanceController uses atomic cluster/store
RebalancePlan
RebalanceBatchPlan
- split storeDefs into currentStoreDefs & finalStoreDefs
- clean up some variable naming

*RoutingStrategy
- marked all override methods with @Override

RouteToAllStrategy
- Implemented getPartitionList/getReplicatingPartitionList/getMasterPartition
- Use a single imaginary parition id ("0") to implement these methods
- This allows StoreRoutingPlan to be constructed for this strategy

AbstractZonedRebalanceTest
- fix proxy put test
d4c314f
@jayjwylie jayjwylie Added more zone expansion test for RebalanceBatchPlan d344725
@jayjwylie jayjwylie Completed unit tests for zone expansion
Added zone expansion unit tests to AbstractZonedRebalanceTest. This required tweaking the helper methods in RebalanceUtils and ClusterTestUtils.
10901f7
@jayjwylie jayjwylie Added junit-rebalance and junit-long-rebalance build targets 92faebd
@jayjwylie jayjwylie Address review of initial RebalanceController work
- Removed 'timeout' option for rebalancing since always setting timeouts
  to infintie (Long.MAX_VALUE) is the recommended practice.

- Cleaned up some TODOs and added some javadoc in response to code
  review from Lei
45903e0
@jayjwylie jayjwylie Fix invalid metadata rate calculation
Fixed the calculation of invalid metadata rate in RebalanceUtils. Also added more/different pretty printing of a rebalance plan.
6a283ae
@jayjwylie jayjwylie Minor TODO and copyright cleanup. c794504
@jayjwylie jayjwylie Fix some bugs I introduced and added more TODOs
Fixed overflow introduced in AdminClient.waitForCompletion by passing in
Long.MAX_VALUE for duration.

Verify cluster store definition in StoreRoutingPlan. This requires
working around existing problems with how system stores are handled (the
store definition is hard-coded for two zones). Left some TODOs about
testing and fixing all of this.

Added TODOs about currentCluster vs interimCluster. Need to tweak
interface to RebalanceController and RebalancePlan to be consistent with
recommended usage (i.e., deploying interimCluster before starting
rebalance).

Minor tweaks to tests based on above changes.
81f0a16
@jayjwylie jayjwylie Fix potential overflow in AdminClient.waitForCompletion 2bf1543
@jayjwylie jayjwylie Fix arg checking typo in RebalanceControllerCLI. d7f81d7
@jayjwylie jayjwylie Add 'proxy pause' between cluster change & rebalance
The proxy pause is a window during which clients can pick up the new
cluster metadata, and servers can establish proxy bridges, before
servers start moving data around for rebalancing. This allows us to
observe the cost of proxying separate from rebalance and allows clients
to pick up new metadata before any data is moved.

Also added some more TODOs about documenting the various rebalanceState
methods...
c9b370e
@jayjwylie jayjwylie Added "progress bar" for rebalance batch plan
Added RebalanceBatchPlanProgressBar
- progress tracking object for each rebalancing batch
- integrated with RebalanceController, AsyncRebalanceTask, and
  (Donor|Stealer)BasedRebalanceTask

Did other general clean up of logging during rebalance to make it
appropriately verbose (more verbose in some places, less verbose in
others...)
4c276b8
@jayjwylie jayjwylie Fixed cluster nomenclature: current, interim, final
Touched a lot of code to normalize variable names, method names, and
comments about clusters:
- current : current cluster in prod
- interim : current cluster + new nodes/zones w empty partitions
- final : final cluster w shuffled partitions and/or populated new
  nodes/zones

Added TODO about voldemort server excepting on invalid cluster xml

Dropped TODOs that were unnecessary
89969de
@jayjwylie jayjwylie Change variable name in RouteToAllStrategy. e876dd1
@jayjwylie jayjwylie Fixed typo. f816900
@jayjwylie jayjwylie Address minor review feedback
Mostly, made variable names more clear

AdminClient
- cleaned up comments & TODO

RebalanceController
- fix subtle bug in when proxyPause invoked

RepartitoinerTest
- changed zone expansion test to cover two use cases: when invoked from
  RepartitoinerCLI with a current cluster versus with an interim
  cluster.
412088d
@jayjwylie jayjwylie Make KeyVersionFetcherCLI ZoneNAry aware ffc9b68
@jayjwylie jayjwylie Fix to print one timestamp per-line from KeyVersionFetcherCLI e40f793
@voldemort Add comments to the tools 3122693
@jayjwylie jayjwylie Make KeyVersionFetcherCLI have generous timeouts 0ef8faf
@jayjwylie jayjwylie Minor fix to KeyVersionFetcherCLI ec9cd1e
@jayjwylie jayjwylie Correctly print RebalanceTask at creation time. 77f3b69
@jayjwylie jayjwylie Fix final-cluster to be required arg of RebalancePlanCLI f55aace
@voldemort Print the utility value 9ee4ebe
@voldemort Write plan to a file inside the output dir aa96bbf
@voldemort Add zone expansion script c1fb3aa
@voldemort Add an option for choosing zone id for shuffle
Add a new option that lets the user choose if they want the
shuffle to operate only on one of the zone ids.
ba371fc
@voldemort Edit script to use the new option 407d4e4
@voldemort Address code review comments 9d9b47e
@voldemort Add rebalance new cluster script f9936c1
@voldemort Copy final cluster xml file to output dir
When the final cluster.xml is generated it is placed in
outputdir/step3 folder. Copy it to outputdir as well.
1807fae
@voldemort Move rebalance scripts to bin directory
Moving the script from top level to the bin directory
803d7dd
@voldemort Add dummy-stores.xml + Update the path in script d2848a1
@abh1nay abh1nay Test for 3 zones 4f135d5
@abh1nay abh1nay Addressed code review comments
added more assertions
check for getZoneNary and getNodeIdForZoneNary  in zone 1 and 2
Added a test for method 'storeRoutingPlan.zoneNAryExists()
9a90156
@vinothchandar vinothchandar Modifying RoutedStoreTest to use Consistent Strategy 3512cff
@abh1nay abh1nay Tests for system store with a 3 zone cluster a758c2a
@abh1nay abh1nay Fixed systemstore constants and got rid of the superfluous zone rep
factor
0c8f0fa
@jayjwylie jayjwylie Fix RebalanceUtils.getLatestCluster.
RebalanceUtils
- changed getLatestCluster to examine contents of cluster.xml on different nodes rather than timestamps since timestamps are incomparable across servers.

AbstractZonedRebalanceTest
- added a specific rebalance test that goes from current -> shuffle and then shuffle -> current. This confirms that rebalance can be invoked repeated times (if need be).
882c01f
Commits on Jun 21, 2013
@voldemort Step 1 : Generate cluster script cleanup d2452a9
@voldemort Step 2: Add logic to handle zoned clusters 5d1e8e3
@voldemort Cleanup the arguments and add sanity check 3944a3c
@voldemort Add separate dummy stores for 2 and 3 zone cluster 3d8eb81
@voldemort Fix comments and typos 155b934
@singhsiddharth singhsiddharth Add 1.3.4 and 1.4.0 release notes 4d72ef4
@voldemort Bump release version 93b9495
Commits on Jun 24, 2013
@zhongjiewu zhongjiewu bug fixes in coordinator 33ea454
Commits on Jun 28, 2013
@jayjwylie jayjwylie Tweaks to PartitionBalance and RebalancePlan
PartitionBalance
- calculate partition-stores per zone.
- This measure provides more context to evaluate the size of any plans to rebalance the cluster.

RebalancePlanCLI
- fix typo in verbose usage message
a26a558
@jayjwylie jayjwylie Initial hack at new rebalance scheduler.
Added RebalanceController.scheduler
- limits each node to participating in a single task as either a stealer or donor.
- randomizes the order in which tasks are attempted to be scheduled
- not a clean implementation, but enough to evaluate.
34e849e
@jayjwylie jayjwylie Tweak new rebalance scheduler
Randomized the order of rebalance tasks in each stealer's list. This
will avoid biasing the rebalance based on the order tasks were
generated.
d7356df
@jayjwylie jayjwylie Addressed review feedback on rebalance scheduler
RebalanceController
- Changed default parallelism to "infinite" since scheduler throttles parallelism
- Added TODOs for cleanup of scheduler
- Added javadoc to document scheduler and its methods
- Catch exceptions, log them, re-throw as VoldemmortRebalancingException

A ton of white space changes due to futzing with eclipse code formatter preferences. Sorry.
f8765a3
@jayjwylie jayjwylie Refactor StoreRoutingPlan. Break Base from rest to expedite construct…
…ion.
606f6fa
@jayjwylie jayjwylie Adding BaseStoreRoutingPlan. bf7daad
@jayjwylie jayjwylie Making StoreRoutingPlan even lighter-weight 5459942
@vinothchandar vinothchandar Update release_notes and build.properties for 1.4.2 release 4a95638