forked from voldemort/voldemort
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge remote-tracking branch 'voldemort/master'
- Loading branch information
Showing
318 changed files
with
28,599 additions
and
7,304 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,8 @@ | ||
| Manifest-Version: 1.0 | ||
| Ant-Version: Apache Ant 1.7.1 | ||
| Created-By: 20.2-b06 (Sun Microsystems Inc.) | ||
| Voldemort-Implementation-Version: 1.3.0 | ||
| Implementation-Title: Voldemort | ||
| Implementation-Version: 0.96 | ||
| Implementation-Version: 1.3.0 | ||
| Implementation-Vendor: LinkedIn | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| This directory contains utility to convert BDB JE data between different versions of Voldemort. | ||
|
|
||
| Need for Conversion | ||
| ------------------- | ||
| Voldemort has been using "sorted duplicates" feature of BDB JE to handle | ||
| conflicting writes to the same key. At the very minimum, the conversion gets | ||
| rid of BDB sorted duplicates support and handles duplicates in the Voldemort | ||
| storage layer itself. The decision was made after months of closely working | ||
| with Oracle JE team, to understand the factors affecting performance. | ||
|
|
||
| Data Formats | ||
| ------------ | ||
| This section describes the data formats themselves. | ||
|
|
||
| 1) Base Format (Base) | ||
| --------------------- | ||
| This is the format used by Voldemort up until 1.1.x, relying on BDB JE for | ||
| duplicate handling | ||
|
|
||
| Disadvantages: | ||
| -- The manner in which BDB JE handles duplicates is not suitable for an | ||
| application with small percent of 2-3 duplicates i.e Voldemort. | ||
| -- Data bloat issue that prevented us from migrating to any higher 4.x version | ||
| to be able to control cache eviction | ||
| -- Incompatible with how duplicates are handled in JE5. | ||
| -- May incur additional locking costs for the "duplicates" subtree | ||
|
|
||
| 2) New duplicate format (NewDup) | ||
| -------------------------------- | ||
| This format is supported from release 1.1.x, where Voldemort storage layer | ||
| handles duplicates and BDB JE version is bumped up to JE 4.1.17 | ||
|
|
||
| Advantages: | ||
| -- Ability to move data off disk. This is very GC friendly, relying on OS page | ||
| cache for the data and using the JVM heap only for index. This is achieved | ||
| by setting "bdb.cache.evictln" server parameter to "true" | ||
| -- Ability to evict data brought into the cache during scans, minimize impact | ||
| on online traffic (Restore, Rebalance, Retention). This is achieved by | ||
| setting "bdb.minimize.scan.impact" to "true" | ||
| -- Thinner storage layer. eg: BdbStorageEngine.put() does not incur the cost | ||
| of an additional delete() | ||
| -- General speed up due to elimination of duplicates | ||
|
|
||
| This format is the minimum requirement to be able to upgrade to 1.1.x & higher | ||
|
|
||
| 3) Partition Scan format (PidScan) | ||
| ---------------------------------- | ||
| This is a super set of 'NewDup' format, supported 1.1.x upwards. In addition to | ||
| eliminating duplicates and upgrading to JE 4.1.17, it adds a 2 byte prefix | ||
| representing the partition id to each key. | ||
|
|
||
| Advantages: | ||
| -- Speed up Restore and Rebalancing linearly to the number of partitions | ||
| actually fetched. (which means much shorter degraded mode performance) | ||
|
|
||
| This is an optional format. You can turn it off, by setting | ||
| bdb.prefix.keys.with.partitionid=false, if you don't like for some reason | ||
|
|
||
| Note : We have not seen the extra 2 bytes cause any overhead to online | ||
| performance | ||
|
|
||
| IMPORTANT: IT IS REQUIRED TO CONVERT TO EITHER 'NewDup' or 'PidScan' TO RUN | ||
| VOLDEMORT WITH BDB, STARTING RELEASE 1.1.x | ||
|
|
||
| Running the Conversion Utility | ||
| ------------------------------ | ||
| The tool provides the ability to convert one database from a source environment | ||
| to a destination environment. You need to run the tool for each of the databases | ||
| or voldemort store you have. You can bring one Voldemort server at a time and | ||
| perform the conversion and bring it up on the appropriate release | ||
|
|
||
| Note: For users running with "bdb.one.env.per.store=false", it means you will | ||
| have to run the tool with the same --src --dest options for each database | ||
| contained. | ||
|
|
||
| In addition to BDB environment locations, the tool needs the cluster.xml to generate | ||
| the partition prefix. | ||
|
|
||
| $./voldemort-convert-bdb.sh --src <Required: Path to source bdb environment> | ||
| --dest <Required: Path to place converted new environment> | ||
| --store <Required: BDB database (voldemort store) name> | ||
| --cluster-xml <Required: Path to cluster.xml> | ||
| --from-format <Required: Format to convert FROM, one of the 3 | ||
| strings 'Base','NewDup','PidScan'> | ||
| --to-format <Required: Format to convert TO, one of the 3 | ||
| strings 'Base','NewDup','PidScan'> | ||
| --je-log-size <Optional: Size in MB for the new JE log files, | ||
| Default:60> | ||
| --btree-nodemax <Optional: Btree fan out, Default: 512> | ||
|
|
||
| We recommend you run the following to move to release 1.1.x & up. | ||
|
|
||
| $./voldemort-convert-bdb.sh --src /path/to/src/env | ||
| --dest /path/to/dest/env | ||
| --store teststore | ||
| --cluster-xml /path/to/cluster/xml | ||
| --from-format Base | ||
| --to-format PidScan | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| @echo off | ||
|
|
||
| REM | ||
| REM Copyright 2013 Carlos Tasada | ||
| REM | ||
| REM Licensed under the Apache License, Version 2.0 (the "License"); | ||
| REM you may not use this file except in compliance with the License. | ||
| REM You may obtain a copy of the License at | ||
| REM | ||
| REM http://www.apache.org/licenses/LICENSE-2.0 | ||
| REM | ||
| REM Unless required by applicable law or agreed to in writing, software | ||
| REM distributed under the License is distributed on an "AS IS" BASIS, | ||
| REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| REM See the License for the specific language governing permissions and | ||
| REM limitations under the License. | ||
| REM | ||
| REM ** This Windows BAT file is not tested with each Voldemort release. ** | ||
|
|
||
| set argC=0 | ||
| for %%a in (%*) do set /a argC+=1 | ||
| if %argC% geq 1 goto :continue | ||
| echo %0 java-class-name [options] | ||
| goto :eof | ||
| :continue | ||
|
|
||
| SET BASE_DIR=%~dp0.. | ||
| SET CLASSPATH=. | ||
|
|
||
| set VOLDEMORT_CONFIG_DIR=%1%/config | ||
|
|
||
| for %%j in ("%BASE_DIR%\dist\*.jar") do (call :append_classpath "%%j") | ||
| for %%j in ("%BASE_DIR%\lib\*.jar") do (call :append_classpath "%%j") | ||
| set CLASSPATH=%CLASSPATH%;"%BASE_DIR%\dist\resources" | ||
| goto :run | ||
|
|
||
| :append_classpath | ||
| set CLASSPATH=%CLASSPATH%;%1 | ||
| goto :eof | ||
|
|
||
| :run | ||
| if "%VOLD_OPTS%" == "" set "VOLD_OPTS=-Xmx2G -server -Dcom.sun.management.jmxremote" | ||
| java -Dlog4j.configuration=%VOLDEMORT_CONFIG_DIR%\log4j.properties %VOLD_OPTS% -cp %CLASSPATH% %* | ||
|
|
||
| endlocal | ||
| :eof |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| @echo off | ||
|
|
||
| REM | ||
| REM Copyright 2013 Carlos Tasada | ||
| REM | ||
| REM Licensed under the Apache License, Version 2.0 (the "License"); | ||
| REM you may not use this file except in compliance with the License. | ||
| REM You may obtain a copy of the License at | ||
| REM | ||
| REM http://www.apache.org/licenses/LICENSE-2.0 | ||
| REM | ||
| REM Unless required by applicable law or agreed to in writing, software | ||
| REM distributed under the License is distributed on an "AS IS" BASIS, | ||
| REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| REM See the License for the specific language governing permissions and | ||
| REM limitations under the License. | ||
| REM | ||
| REM ** This Windows BAT file is not tested with each Voldemort release. ** | ||
|
|
||
| SET BASE_DIR=%~dp0.. | ||
|
|
||
| call "%BASE_DIR%/bin/run-class.bat" voldemort.VoldemortAdminTool %* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| #!/bin/bash | ||
| # | ||
| # Copyright 2008-2012 LinkedIn, Inc | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| base_dir=$(dirname $0)/.. | ||
|
|
||
| for file in $base_dir/dist/*.jar; | ||
| do | ||
| CLASSPATH=$CLASSPATH:$file | ||
| done | ||
|
|
||
| for file in $base_dir/lib/*.jar; | ||
| do | ||
| CLASSPATH=$CLASSPATH:$file | ||
| done | ||
|
|
||
| for file in $base_dir/contrib/*/lib/*.jar; | ||
| do | ||
| CLASSPATH=$CLASSPATH:$file | ||
| done | ||
|
|
||
| CLASSPATH=$CLASSPATH:$base_dir/dist/resources | ||
|
|
||
| JVM_OPTS="-server -Xms5g -Xmx5g -XX:NewSize=1024m -XX:MaxNewSize=1024m -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2" | ||
|
|
||
| java -Dlog4j.configuration=src/java/log4j.properties $JVM_OPTS -cp $CLASSPATH voldemort.store.bdb.dataconversion.BdbConvertData $@ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| @echo off | ||
|
|
||
| REM | ||
| REM Copyright 2013 Carlos Tasada | ||
| REM | ||
| REM Licensed under the Apache License, Version 2.0 (the "License"); | ||
| REM you may not use this file except in compliance with the License. | ||
| REM You may obtain a copy of the License at | ||
| REM | ||
| REM http://www.apache.org/licenses/LICENSE-2.0 | ||
| REM | ||
| REM Unless required by applicable law or agreed to in writing, software | ||
| REM distributed under the License is distributed on an "AS IS" BASIS, | ||
| REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| REM See the License for the specific language governing permissions and | ||
| REM limitations under the License. | ||
| REM | ||
| REM ** This Windows BAT file is not tested with each Voldemort release. ** | ||
|
|
||
| SET BASE_DIR=%~dp0.. | ||
|
|
||
| call "%BASE_DIR%/bin/run-class.bat" voldemort.performance.benchmark.Benchmark %* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| @echo off | ||
|
|
||
| REM | ||
| REM Copyright 2013 Carlos Tasada | ||
| REM | ||
| REM Licensed under the Apache License, Version 2.0 (the "License"); | ||
| REM you may not use this file except in compliance with the License. | ||
| REM You may obtain a copy of the License at | ||
| REM | ||
| REM http://www.apache.org/licenses/LICENSE-2.0 | ||
| REM | ||
| REM Unless required by applicable law or agreed to in writing, software | ||
| REM distributed under the License is distributed on an "AS IS" BASIS, | ||
| REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| REM See the License for the specific language governing permissions and | ||
| REM limitations under the License. | ||
| REM | ||
| REM ** This Windows BAT file is not tested with each Voldemort release. ** | ||
|
|
||
| SET BASE_DIR=%~dp0.. | ||
|
|
||
| call "%BASE_DIR%/bin/run-class.bat" voldemort.client.rebalance.RebalanceCLI %* |
Oops, something went wrong.