Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

A Content Storage Layer over a Sparse Map (eg Column DB)

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 drivers
Octocat-spinner-32 extensions
Octocat-spinner-32 parent
Octocat-spinner-32 src
Octocat-spinner-32 .gitignore
Octocat-spinner-32 COMMITTERS
Octocat-spinner-32 CONTRIBUTING
Octocat-spinner-32 LICENSE
Octocat-spinner-32 NOTICE
Octocat-spinner-32 README.textile
Octocat-spinner-32 pom.xml
README.textile

Map Content System.

Rational

In the Q1 release of Nakamura we had major scalability and concurrency problems caused mainly by our use cases for a content

store not being closely aligned with those of Jackrabbit. We were not able to work around those problems and although we did manage
to release the code, its quite clear that in certain areas Jackrabbit wont work for us. This should not reflect badly on Jackrabbit,
but it is a realization that our use cases are not compatible with Jackrabbit when exposed to scale.

This code base is a reaction to that. It aims to be really simple, completely concurrent with no synchronization and designed to scale

linearly with the number of cores and number of servers in a cluster. To do this it borrows some of the concepts from JCR at a very
abstract level, but is making a positive effort and selfish effort to only provide those things that we absolutely need to have.

This code provides User, Group, Access Control and Content functionality using a sparse Map as a storage abstraction. The Implementation works on manipulating sparse objects in the Map with operations like get, insert and delete, but

has no understanding of the underlying implementation of the storage mechanism.

At the moment we have 3 storage mechanisms implemented, In Memory using a HashMap, Cassandra and JDBC capable of doing sharded storage, The approach should

work on any Column Store (Dynamo, BigTable, Riak, Voldomort, Hbase etc).

Query support is provided by finder messages that use index table written on update. Caching support is via an interface allowing external providers. In Nakamura there is an EhCache implementation and it would be a relatively simple task to write an Infinispan version. Transactions are supported, if supported by the underlying implementation of the storage, otherwise all operations are BASIC, non Atomic and immediate in nature. Search is provided in the form of a companion project that uses SolrJ 4 The JDBC Driver has configuration files for Derby, MySQL, Oracle, PostgreSQL.

Backlog

  1. Implement Infinispan StorageClient cache (Read or Write Through)

Completed Backlog

  1. Check all byte[] usage and limit to only real bodies. (14/11/2010) no byte[] are used for content bodies.
  2. Replace all byte[] usage with InputStreams or a resetable holder, pushing down into the StorageClient. (14/11/2010)
  3. Provide Read Cache implementation of StorageClient that chains to a real storage client (28/11/2010, CachedManager)
  4. Provide Write Through Cache implementation of StorageClient that chains to a real storage client, (wont fix)
  5. Provide Scoped (as in Transaction Scoped) implementation of StorageClient that chains to a real storage client (wont fix since there which Jackrabbit in place there is no guarentee that the session you see on the request is the session that should be committed)
  6. Do scalability testing on MySQL and Cassandra (done 22/11/2010)
  7. Implement SparseMapUserManager and related classes in th server bundle in Sling. (done 28/11/2010)

Tests

Memory

All performed on a MackBook Pro which is believed to have 4 cores.
Add a user, 1 – 10 threads. Storage is a Concurrent Hash Map. Assuming the Concurrent Hash Map is 100% concurrent, this test
tests the code base for concurrent efficiency.

Threads Time(s) Throughput Throughput per thread Speedup Concurrent Efficiency
1 0.46 2188 2188 1 100%
2 0.18 5495 2747 2.51 126%
3 0.05 21739 7246 9.93 331%
4 0.14 7143 1786 3.26 82%
5 0.1 10309 2062 4.71 94%
6 0.25 4049 675 1.85 31%
7 0.05 20408 2915 9.33 133%
8 0.03 33333 4167 15.23 190%
9 0.25 4082 454 1.87 21%
10 0.14 7042 704 3.22 32%

Throughput is users added per second.

JDBC

Same as above, using a local MySQL Instance.

Threads Time(s) Throughput Throughput per thread Speedup Concurrent Efficiency
1 12.19 82 82 1 100%
2 9.65 104 52 1.26 63%
3 11.18 89 30 1.09 36%
4 15.89 63 16 0.77 19%
5 9.65 104 21 1.26 25%
6 16.73 60 10 0.73 12%
7 21.76 46 7 0.56 8%
8 13.96 72 9 0.87 11%
9 10.17 98 11 1.2 13%
10 11.47 87 9 .06 11%

Cassandra

Using an untuned OOTB Cassandra instance running on the same box as the test, fighting for processor Cores.

Threads Time(s) Throughput Throughput per thread Speedup Concurrent Efficiency
1 1.14 873 873 1 100%
2 0.65 1520 760 1.74 87%
3 0.44 2227 742 2.55 85%
4 0.46 2146 536 2.46 61%
5 0.43 2320 464 2.66 53%
6 0.3 3257 543 3.73 62%
7 0.28 3521 503 4.03 58%
8 0.28 3546 443 4.06 51%
9 0.34 2890 321 3.31 37%
10 0.37 2703 270 3.09 31%

Throughput is users added per second.

So far it looks like the code is concurrent, but MySQL is considerably slower than Cassandra or Memory. Below the Fighting for cores
the box doesn’t have enough CPUs to support the DB if present and the code.

Contributions, Patches, and License.

The code in this code base is © Timefields Ltd and licensed to the Sakai Foundation under a Apache 2 Software License. Before making
a contribution by means of a patch or otherwise please ensure that you read and understand the terms under which a patch or contribution
will be accepted, outlined in the NOTICES file. All patches and contributions are made under those terms with no exceptions.

I am sorry if this sounds a bit legal, but I want to be able to always license this software to the Sakai Foundation under an Apache 2 license
and so I have to insist that no contributions are made that would prevent that from happening. I cani’t ask everyone who submits a patch to sign a
legal document, so this is the next best thing. If you have a problem with this approach, please email me and we can try and work it out.

BTW, IANAL :)

Something went wrong with that request. Please try again.