Permalink
Browse files

Scribe dynamic bucket id to network store configuration

Summary:
this diff added feature such that scribe will be able to work against a
zookeeper managed network cluster.  Instead of embedding zookeeper into scribe,
we added a bucketupdater thrift interface that all zookeeper managed hosts
should support that return a bid to host:port mapping.

added bucketupdate test to testsuites.php.

tested with testsuite.

Test Plan:
bucketupdater test case added to testsuites.php.
"
// testing bucket store updater.
// Setup:
// 1. two scribe servers.  one running on port 1465 and using conf file
//    scribe.conf.bucketupdater.server1, and the other running on port 1466
//    and using conf file scribe.conf.bucketupdater.server2.
//    scribe server that configured by scribe.bucketupdater.server1
//    writes scribe messages to /tmp/scribetest_/bucketupdater/server1
//    and scribe server that configured by scribe.bucketupdater.server2
//    writes scribe messages to /tmp/scribetest_/bucketupdater/server2
// 2. bidupdater server which implements bucket updater interface.
//    It reads mappings from a local file bidmap which is symbolic
//    linked to bidmap.1 or bidmap.2.  bidmap.1 maps bucket 1 to
//    scribe server that writes to /tmp/scribetest_/bucketupdater/server1
//    and bucket 2 to scribe server that writes to
/tmp/scribetest_/bucketupdater/server2.
//    bidmap.2 does the opposite.
//
// Test setup:
// 1. launch scribe -p 1465 scribe.conf.bucketupdater.srever1
// 2. launch scribe -p 1466 scribe.conf.bucketupdater.server2
// 3. ln -sf bidmap.1 bidmap.
// 4. launch bidupdater: bidupdater bidmap
// 5. scribe -p 1463 scribe.conf.bucketupdater which use bidupdater
//    to dynamically configure bucket store: bucketupdater.
// 6. send two message, one with bucket id 1 and the other with bucket id 2
//    to scribe running on 1463.  Check that the messages are
//    in /tmp/scribetest_/bucketupdater/server1/bucketupdater/bucket1
//    and /tmp/scribetest_/bucketupdater/server2/bucketupdater/bucket2
//    respectively.
// 7. ln -sf bidmap.2 bid
// 8. wait for 15 seconds
// 9. send another two messages, one with bucket 1 and the other bucket 2
//    Check that messages are in the reverse of 6.
"

DiffCamp Revision: 108360
Reviewed By: pkhemani
Commenters: groys, zshao
CC: agiardullo, jsong, pkhemani, groys, scribe-dev@lists
Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28165 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
  • Loading branch information...
1 parent 17fe373 commit 9c231ed2f820bdc64a0c1e604e034cfd87d9ce59 groys committed with groys May 20, 2010
View
@@ -0,0 +1,37 @@
+#!/usr/local/bin/thrift --cpp --php
+
+## Copyright (c) 2009- Facebook
+##
+## Licensed under the Apache License, Version 2.0 (the "License");
+## you may not use this file except in compliance with the License.
+## You may obtain a copy of the License at
+##
+## http://www.apache.org/licenses/LICENSE-2.0
+##
+## Unless required by applicable law or agreed to in writing, software
+## distributed under the License is distributed on an "AS IS" BASIS,
+## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+## See the License for the specific language governing permissions and
+## limitations under the License.
+##
+## See accompanying file LICENSE or visit the Scribe site at:
+## http://developers.facebook.com/scribe/
+
+namespace cpp scribe.thrift
+namespace java com.facebook.infrastructure.service
+
+// BucketStoreMapping service exception
+exception BucketStoreMappingException {
+ 1: string message;
+ 2: i32 code;
+}
+
+struct HostPort {
+ 2: string host,
+ 3: i32 port
+}
+
+service BucketStoreMapping {
+ // given a category, return a list of HashCodeToNetworkStore mappings
+ map<i32, HostPort> getMapping(1: string category) throws (1: BucketStoreMappingException e);
+}
Oops, something went wrong.

0 comments on commit 9c231ed

Please sign in to comment.