Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
SOLR-13101 : Shared storage support in SolrCloud #864
This PR is being opened to expose the code for integrating SolrCloud with a shared blobstore. This is a work in progress - we are hoping to open this up to community discussion and gather feedback.
Solr + Blobstore
This repo introduces a new framework which allows SolrCloud to integrate with an external (typically cloud-based) blobstore. Instead of maintaining a copy of the index on each Solr host, replicating updates to peers, and using a transaction log to maintain consistent ordered updates, Solr hosts will push and pull cores to/from this external store.
TL;DR: For now, SolrCloud can be configured to use blobstore at a collection level. Collections backed by blobstore use a new SHARED replica type. When a Solr node makes an update request to a shared shard, it indexes locally and then pushes the change through to a shared blobstore. Zookeeper manages index versioning and provides a source of truth in the case of concurrent writes. Solr nodes in a cluster will no longer use peer-to-peer replication, and instead will pull updates directly from the shared blobstore.
Please note that this project is a work in progress, and is by no means production-ready. This code is being published early get feedback, which we will incorporate in future work.
In order to modularize these changes and maintain existing functionality, most of the blobstore-related code is isolated to the solr/core/src/java/org/apache/solr/store/blob directory. However, there some key integration touchpoints in HttpSolrCall#init, DistributedZkUpdateProcessor, and CoreContainer#load. These classes all have special handling for blobstore-based shards.
Pulling from Blobstore
Core pulls are, for the most part, asynchronous. When a replica is queried, it enqueues a pull from blobstore but doesn’t wait for the pull to complete before it executes the query, unless the replica is missing a copy of that core altogether. If your operation requires that local cores are in-sync with blobstore, use the method BlobStoreUtils#syncLocalCoreWithSharedStore.
A more in-depth walkthrough of the pull code:
Pushing to Blobstore
This happens synchronously. On every local commit, we push to blobstore and only ack that the update was successful when it is committed both locally and in the shared store.
A more in-depth walkthrough of the push code:
Resolving Local and Blobstore
The SharedStoreResolutionUtil handles resolving diffs between the Solr node’s local copy of a core and the copy in blobstore. It does so by pulling the metadata for the core from blobstore (BlobCoreMetadata), comparing against the local metadata (ServerSideMetadata), and creating a list of segments to push or pull.
Only the leader node can push updates to blobstore. Because a new leader can be elected at any time, there is still a possibility for race conditions on writes to blobstore. In order to maintain a consistent global view of the latest version of a core, we keep version data in Zookeeper.
Zookeeper stores this version data as a random string called metadataSuffix. When a SolrCloud node makes an update request, it first pushes the files to blobstore and then makes a conditional update to the metadataSuffix variable. If Zookeeper rejects the conditional update, the update request fails, and the failure is propagated back to the client.
This communication with Zookeeper is coordinated in the SharedShardMetadataController. The SharedShardMetadataController belongs to the Overseer (i.e. the leader replica).
Try it yourself
If you want to try this out locally, you can start up SolrCloud with the given blobstore code. The code will default to using the local blobstore client, with /tmp/BlobStoreLocal as the blobstore directory (see LocalStorageClient). You can create a shared collection through the Solr admin UI by setting “shared store based” to true.
Note: if you want to try testing with the S3StorageClient, you need to store a valid S3 bucket name and credentials as environment variables (see S3StorageClient#AmazonS3Configs).
Find tests in the test directory (
Please review the following and check all that apply:
SOLR-13101: Added configuration for Collections backed by shared storage and a new SHARE replica type W-6020845
Updating DistributedZkUpdateProcessorTest to include cases for shared and non-shared replica types. Also added a method to the SolrCloudTestCase that allows accessing the core container of a given node so that we can actually access a specific core in tests.
Jenkinsfile with build and test steps commented out for the time being. To unblock blob work.
Wait for missing core to be pulled when queried, and return results. Also skipping recovery for SHARED replicas.
* Update s3 client info * update S3 client * Add comments * Fix related files * update s3 client code * delete test data * Address review comments and default to local filesystem when storage provider isnt defined * fix PR comments
eribeiro left a comment •
Congratulations to Ilan Ginzburg, Yonik Seeley and team for one of the coolest talks I have seen at Activate 2019. Good to know that you are opening this WIP project to the community. :) I took the liberty to give a first time, first glance review. Please, let me know if you find it worth or not, so that I can know if I am going to do a further, in depth look at the code. Again, congrats!
OK folks, I've created copy of this PR as a branch in the ASF repo for future work: