Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 139 lines (109 sloc) 7.865 kB
a31b20a [SLINGCSCF-2004] [SLINGCSCF-2073] Remove Yahoo dependent code & Integ…
Francisco Perez-Sorrosal authored
1 Omid
2 ====
2e435f5 @dgomezferro Omid project
dgomezferro authored
3
a31b20a [SLINGCSCF-2004] [SLINGCSCF-2073] Remove Yahoo dependent code & Integ…
Francisco Perez-Sorrosal authored
4 The Omid project provides transactional support for key-value stores using Snapshot Isolation. Omid stands for Optimistically transactional Management in Datasources. HBase is the only datastore currently supported, though adaption to any datastore that provides multiple versions per cell should be straightforward.
2e435f5 @dgomezferro Omid project
dgomezferro authored
5
a31b20a [SLINGCSCF-2004] [SLINGCSCF-2073] Remove Yahoo dependent code & Integ…
Francisco Perez-Sorrosal authored
6 There are 3 components in OMID;
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
7 * The Transaction Status Oracle (TSO), which assigns transaction timestamps and resolves conflicts between transactions.
8 * The commit table which stores a mapping from start timestamp to commit timestamp
9 * Shadow cells, which are written alongside data cells in the datastore to allow client to resolve reads without consulting the commit table.
2e435f5 @dgomezferro Omid project
dgomezferro authored
10
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
11 To start a transaction, the client requests a start timestamp from the TSO. It then writes any data cells it wishes to write (i.e. the write set) with the start timestamp as the version of the data cell. To commit the transaction, the client sends the write set to the TSO, which will check for conflicts. If there is no conflict, the TSO assigns a commit timestamp to the transaction, and writes the mapping of the start timestamp to commit timestamp to the commit table. Once the mapping has been persisted, the commit timestamp is returned to the client. On receiving the commit timestamp from the server, the client updates the shadow cells for the write set and clears the transaction from the commit table.
2e435f5 @dgomezferro Omid project
dgomezferro authored
12
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
13 To read a cell transactionally, the client first checks if the cell has a shadow cell. If the commit timestamp of the shadow cell is lower than the start timestamp of the reading transaction, then the client can "see" the cell. If there is no shadow cell, the commit table is consulted to find the commit timestamp of the cell. If the commit timestamp does not exist in the commit table, then the cell is assumed to below to an aborted transaction.
39ea42c README improved
Francisco Perez-Sorrosal authored
14
a31b20a [SLINGCSCF-2004] [SLINGCSCF-2073] Remove Yahoo dependent code & Integ…
Francisco Perez-Sorrosal authored
15 There are currently two implementations of the commit table, a HBase implementation and an In-Memory implementation. The In-Memory implementation gives no persistence guarantee and is only useful for benchmarking the TSO.
39ea42c README improved
Francisco Perez-Sorrosal authored
16
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
17 Quickstart
18 ----------
19 Clone the repository and build the TSO package:
39ea42c README improved
Francisco Perez-Sorrosal authored
20
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
21 $ git clone git@git.corp.yahoo.com:scalable-computing/omid.git
22 $ cd omid
23 $ mvn clean install assembly:single
2e435f5 @dgomezferro Omid project
dgomezferro authored
24
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
25 This will generate a binary package containing all dependencies for the TSO in tso-server/target/tso-server-<VERSION>-bin.tar.gz.
7ff7a06 README.md improved
Francisco Perez-Sorrosal authored
26
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
27 Extract this package on your server
2e435f5 @dgomezferro Omid project
dgomezferro authored
28
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
29 $ tar zxvf tso-server-<VERSION>-bin.tar.gz
30 $ cd tso-server-<VERSION>
2e435f5 @dgomezferro Omid project
dgomezferro authored
31
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
32 Ensure that the setting for hbase.zookeeper.quorum in conf/hbase-site.xml points to your zookeeper instance, and create the commit and timestamp tables.
33
34 $ bin/omid.sh create-hbase-commit-table -numSplits 16
35 $ bin/omid.sh create-hbase-timestamp-table
2e435f5 @dgomezferro Omid project
dgomezferro authored
36
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
37 Then start the TSO.
2e435f5 @dgomezferro Omid project
dgomezferro authored
38
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
39 $ bin/omid.sh tso
2e435f5 @dgomezferro Omid project
dgomezferro authored
40
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
41 By default the tso listens on port 54758.
2e435f5 @dgomezferro Omid project
dgomezferro authored
42
daf4196 Update README.md
Francisco Perez-Sorrosal authored
43 HBase Client usage
44 ------------------
39ea42c README improved
Francisco Perez-Sorrosal authored
45
1377559 Added capability to connect to HBase securely
Sameer Paranjpye authored
46 The client interfaces for OMID2 are _TTable_ and _TransactionManager_. A builder is provided in the
47 _HBaseTransactionManager_ class in order to get the TransactionManager interface, which is used for creating and
48 committing transactions. TTable can be used for putting, getting and scanning entries in a HBase table. TTable's
49 interface is similar to the standard _HTableInterface_, and only requires passing the transactional context as a
50 first parameter in the transactional aware methods (e.g. _put(Transaction tx, Put put)_)
51 _These interfaces will likely change slightly in future._
52
16b8329 Update maven dependency info in README
Sameer Paranjpye authored
53 To run this example, make sure _core-site.xml_ and _hbase-site.xml_ for your HBase cluster are present in
1377559 Added capability to connect to HBase securely
Sameer Paranjpye authored
54 your classpath. You will need to set _tso.host_ and _tso.port_ appropriately. Also, you will need to create a hbase
55 table "EXAMPLE_TABLE", with column family "EXAMPLE_CF", and with TTL disabled and maxVersions set to Integer.MAX_VALUE.
56 This example assumes non-secure communication with HBase. If your HBase cluster is secured with Kerberos, you will
57 need to use the `UserGroupInformation` API to log in securely.
2e435f5 @dgomezferro Omid project
dgomezferro authored
58
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
59 ```java
60 import org.apache.hadoop.conf.Configuration;
61 import org.apache.hadoop.hbase.HBaseConfiguration;
62 import org.apache.hadoop.hbase.client.Put;
63 import org.apache.hadoop.hbase.util.Bytes;
daf4196 Update README.md
Francisco Perez-Sorrosal authored
64 import com.yahoo.omid.transaction.HBaseTransactionManager;
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
65 import com.yahoo.omid.transaction.TTable;
66 import com.yahoo.omid.transaction.Transaction;
67 import com.yahoo.omid.transaction.TransactionManager;
2e435f5 @dgomezferro Omid project
dgomezferro authored
68
daf4196 Update README.md
Francisco Perez-Sorrosal authored
69
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
70 public class Example {
71 public static void main(String[] args) throws Exception {
72 Configuration conf = HBaseConfiguration.create();
2e435f5 @dgomezferro Omid project
dgomezferro authored
73
daf4196 Update README.md
Francisco Perez-Sorrosal authored
74 TransactionManager tm = HBaseTransactionManager.newBuilder()
75 .withConfiguration(conf)
76 .build();
1377559 Added capability to connect to HBase securely
Sameer Paranjpye authored
77
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
78 TTable tt = new TTable(conf, "EXAMPLE_TABLE");
79 byte[] exampleRow1 = Bytes.toBytes("EXAMPLE_ROW1");
80 byte[] exampleRow2 = Bytes.toBytes("EXAMPLE_ROW2");
81 byte[] family = Bytes.toBytes("EXAMPLE_CF");
82 byte[] qualifier = Bytes.toBytes("foo");
83 byte[] dataValue1 = Bytes.toBytes("val1");
84 byte[] dataValue2 = Bytes.toBytes("val2");
2e435f5 @dgomezferro Omid project
dgomezferro authored
85
daf4196 Update README.md
Francisco Perez-Sorrosal authored
86 Transaction tx = tm.begin();
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
87 Put row1 = new Put(exampleRow1);
88 row1.add(family, qualifier, dataValue1);
daf4196 Update README.md
Francisco Perez-Sorrosal authored
89 tt.put(tx, row1);
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
90 Put row2 = new Put(exampleRow2);
91 row2.add(family, qualifier, dataValue2);
daf4196 Update README.md
Francisco Perez-Sorrosal authored
92 tt.put(tx, row2);
93 tm.commit(tx);
1377559 Added capability to connect to HBase securely
Sameer Paranjpye authored
94
f2867ac @ivankelly Updated readme for OMID2
ivankelly authored
95 tt.close();
96 tm.close();
97 }
98 }
daf4196 Update README.md
Francisco Perez-Sorrosal authored
99 ```
978aac4 Yinst packaging for Omid compactor
Sameer Paranjpye authored
100
101 HBase Co-processor Usage
102 ------------------------
a31b20a [SLINGCSCF-2004] [SLINGCSCF-2073] Remove Yahoo dependent code & Integ…
Francisco Perez-Sorrosal authored
103 Omid includes an HBase co-processor that operates during compactions (minor and major) and performs
978aac4 Yinst packaging for Omid compactor
Sameer Paranjpye authored
104 data cleanup. Specifically, it does the following:
105 * Cleans up garbage data from aborted transactions
106 * Purges deleted cells. Omid deletes work by placing a special tombstone marker in cells. The compactor
107 detects these and actually purges data when it is safe to do so (i.e. when there are no committable transactions
108 that may read the data).
109 * 'Heals' committed cells for which the writer failed to write shadow cells.
110
111 To deploy the co-processor, the co-processor jar must be placed in a location (typically on HDFS) that is accessible
112 by HBase region servers. The co-processor may then be enabled on a transactional table by the following steps:
113 * Add a co-processor specification to the table via a "co-processor attribute". The co-processor spec may (and usually will)
114 also include the name of the Omid commit table.
115 * Add an "OMID_ENABLED => true" flag to any column families which the co-processor should work on.
116 * Disable and re-enable the table.
117
118 Sample co-processor enabled table:
119 ```
120 'omid_enabled_table', {TABLE_ATTRIBUTES =>
121 {coprocessor$1 => 'hdfs:///hbase/co_processor/omid/omid2-hbase-coprocessor-2.2.11.jar|
122 com.yahoo.omid.transaction.OmidCompactor|1001|
123 omid.committable.tablename=ns:OMID_COMMIT_TABLE},
124 {NAME => 'n', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
125 COMPRESSION => 'LZO', VERSIONS => '2147483647', TTL => 'FOREVER', MIN_VERSIONS => '0',
126 KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'true', BLOCKCACHE => 'true'},
127 {NAME => 's', DATA_BLOCK_ENCODING => 'PREFIX', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
128 COMPRESSION => 'LZO', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0',
129 KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true',
130 METADATA => {'OMID_ENABLED' => 'true'}}
131 ```
2f90e58 Add documentation
Francisco Perez-Sorrosal authored
132
133 Module Dependencies
134 -------------------
27a8b58 [SLINGCSCF-2004] Add Apache License v2 & Copyright headers
Francisco Perez-Sorrosal authored
135 ![Module dependencies](https://git.corp.yahoo.com/scalable-computing/omid/raw/master/doc/images/ModuleDependencies.png "Module dependencies")
136
137 License
138 -------
139 Code licensed under the Apache License Version 2.0. See LICENSE file for terms.
Something went wrong with that request. Please try again.