Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

basic structure

  • Loading branch information...
commit 44bdfb1ffba4472ce21754e78ccfca00b5a0c726 0 parents
Nick Kallen authored
Showing with 1,274 additions and 0 deletions.
  1. +202 −0 LICENSE
  2. +5 −0 TODO
  3. +71 −0 ant/bootstrap.xml
  4. +20 −0 ant/clean.xml
  5. +225 −0 ant/compile.xml
  6. +35 −0 ant/docs.xml
  7. +129 −0 ant/package.xml
  8. +55 −0 ant/prepare.xml
  9. +56 −0 ant/test.xml
  10. +19 −0 build.xml
  11. +6 −0 config/development.conf
  12. +5 −0 config/production.conf
  13. +4 −0 config/test.conf
  14. +42 −0 ivy/ivy.xml
  15. +30 −0 ivy/ivysettings.xml
  16. BIN  libs/gizzard-1.0.jar
  17. +9 −0 src/main/scala/com/twitter/rowz/CopyManager.scala
  18. +10 −0 src/main/scala/com/twitter/rowz/ForwardingManager.scala
  19. +20 −0 src/main/scala/com/twitter/rowz/Hash.scala
  20. +68 −0 src/main/scala/com/twitter/rowz/Main.scala
  21. +57 −0 src/main/scala/com/twitter/rowz/Rowz.scala
  22. +6 −0 src/main/scala/com/twitter/rowz/Shard.scala
  23. +6 −0 src/main/scala/com/twitter/rowz/jobs/Create.scala
  24. +6 −0 src/main/scala/com/twitter/rowz/jobs/Delete.scala
  25. +22 −0 src/main/scala/com/twitter/rowz/thrift/RowzService.scala
  26. +22 −0 src/main/thrift/Rowz.thrift
  27. +127 −0 src/scripts/rowz.sh
  28. +12 −0 src/test/scala/com/twitter/rowz/SampleSpec.scala
  29. +5 −0 src/test/scala/com/twitter/rowz/TestRunner.scala
202 LICENSE
@@ -0,0 +1,202 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
5 TODO
@@ -0,0 +1,5 @@
+- Move shardRepo config into gizzard:
+ shardRepository += ("com.twitter.rowz.SqlShard" -> new SqlShardFactory(queryEvaluatorFactory, config))
+ shardRepository += ("com.twitter.gizzard.ReadOnlyShard" -> new gizzard.ReadOnlyShardFactory)
+ shardRepository += ("com.twitter.gizzard.BlockedShard" -> new gizzard.BlockedShardFactory)
+- Make ThrottledLogger, etc. part of Gizzard
71 ant/bootstrap.xml
@@ -0,0 +1,71 @@
+<project xmlns:ivy="antlib:org.apache.ivy.ant">
+
+ <!-- defaults for all projects -->
+ <property name="source.dir" value="${basedir}/src/main" />
+ <property name="test.source.dir" value="${basedir}/src/test" />
+ <property name="target.dir" value="${basedir}/target" />
+
+ <property environment="env" />
+
+ <property name="ivy.install.version" value="2.1.0-rc2" />
+ <property name="ivy.jar.dir" value="${user.home}/.ivy2" />
+ <property name="ivy.jar.file" value="${ivy.jar.dir}/ivy-${ivy.install.version}.jar" />
+ <property name="jsch.install.version" value="0.1.29" />
+ <property name="jsch.jar.file" value="${ivy.jar.dir}/jsch-${jsch.install.version}.jar" />
+
+ <!--
+ download ivy from the web site so that it can be used without being
+ installed. if the file has already been downloaded, we use a rename
+ trick to avoid hitting the website again. (that would be annoying
+ when building offline.)
+ -->
+ <target name="download-ivy" unless="skip.download">
+ <mkdir dir="${ivy.jar.dir}"/>
+ <condition property="ivy.url" value="file:${ivy.jar.file}">
+ <available file="${ivy.jar.file}" />
+ </condition>
+ <property name="ivy.url" value="http://repo1.maven.org/maven2/org/apache/ivy/ivy/${ivy.install.version}/ivy-${ivy.install.version}.jar" />
+ <get src="${ivy.url}" dest="${ivy.jar.file}.download" usetimestamp="true" />
+ <move file="${ivy.jar.file}.download" tofile="${ivy.jar.file}" />
+
+ <condition property="jsch.url" value="file:${jsch.jar.file}">
+ <available file="${jsch.jar.file}" />
+ </condition>
+ <property name="jsch.url" value="http://repo1.maven.org/maven2/jsch/jsch/${jsch.install.version}/jsch-${jsch.install.version}.jar" />
+ <get src="${jsch.url}" dest="${jsch.jar.file}.download" usetimestamp="true" />
+ <move file="${jsch.jar.file}.download" tofile="${jsch.jar.file}" />
+ </target>
+
+ <!-- import ivy's ant tasks -->
+ <target name="install-ivy" depends="download-ivy">
+ <path id="ivy.lib.path">
+ <fileset dir="${ivy.jar.dir}" includes="ivy-${ivy.install.version}.jar jsch-${jsch.install.version}.jar"/>
+ </path>
+ <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpathref="ivy.lib.path" />
+ </target>
+
+ <!-- define filename-friendly names for the OS variants -->
+ <condition property="os.libsname" value="osx">
+ <os name="Mac OS X" />
+ </condition>
+ <condition property="os.jni.ext" value="jnilib">
+ <os name="Mac OS X" />
+ </condition>
+ <condition property="os.libsname" value="linux">
+ <os name="Linux" />
+ </condition>
+ <condition property="os.jni.ext" value="so">
+ <os name="Linux" />
+ </condition>
+
+ <!-- where to look for the ivy config -->
+ <property name="ivy.dep.file" value="${basedir}/ivy/ivy.xml" />
+ <property name="ivy.settings.file" value="${basedir}/ivy/ivysettings.xml" />
+
+ <import file="clean.xml" />
+ <import file="prepare.xml" />
+ <import file="compile.xml" />
+ <import file="test.xml" />
+ <import file="docs.xml" />
+ <import file="package.xml" />
+</project>
20 ant/clean.xml
@@ -0,0 +1,20 @@
+<project xmlns:ivy="antlib:org.apache.ivy.ant">
+
+ <target name="clean" depends="init" description="erase built files and targets">
+ <delete dir="${target.dir}" />
+ <delete dir="${dist.dir}" />
+ <!-- i dont think this is really a good idea: -->
+ <!-- delete dir="${ivy.jar.dir}/cache/${ivy.organisation}" /-->
+ </target>
+
+ <target name="clean-ivy" depends="prepare" description="erase ivy cache of downloaded packages">
+ <ivy:cleancache />
+ </target>
+
+ <target name="clean-jni" depends="prepare" description="clean out any built jni targets" if="build.jni">
+ <ant dir="src/main/jni" target="clean" />
+ </target>
+
+ <target name="distclean" depends="clean, clean-jni, clean-ivy" />
+
+</project>
225 ant/compile.xml
@@ -0,0 +1,225 @@
+<project xmlns:ivy="antlib:org.apache.ivy.ant">
+
+ <property name="thrift.source.dir" value="${source.dir}/thrift" />
+ <property name="thrift.target.dir" value="${target.dir}/gen-java" />
+
+
+ <!-- generate thrift stubs if necessary -->
+
+ <macrodef name="generate-thrift">
+ <sequential>
+ <pathconvert property="thrift.gen.path">
+ <path location="${ivy.extra.thriftpackage}" />
+ <unpackagemapper from="${basedir}/*" to="${thrift.target.dir}/*" />
+ </pathconvert>
+ <mkdir dir="${thrift.target.dir}" />
+ <property name="thrift.bin" value="thrift" />
+ <apply executable="${thrift.bin}" dest="${thrift.gen.path}" failonerror="true"
+ skipemptyfilesets="true" verbose="true">
+ <arg value="--gen" />
+ <arg value="java" />
+ <arg value="--gen" />
+ <arg value="rb" />
+ <arg value="-o" />
+ <arg value="${target.dir}" />
+ <arg value="" />
+ <fileset dir="${thrift.source.dir}" includes="**/*.thrift" />
+ <globmapper from="*.thrift" to="*.java" />
+ </apply>
+ </sequential>
+ </macrodef>
+
+ <macrodef name="compile-generated-thrift">
+ <sequential>
+ <javac srcdir="${thrift.target.dir}" destdir="${target.dir}/classes" deprecation="on">
+ <classpath>
+ <path refid="deps.path" />
+ </classpath>
+ <include name="**/*.java" />
+ </javac>
+ </sequential>
+ </macrodef>
+
+ <target name="compile-thrift" if="build.thrift">
+ <generate-thrift />
+ <compile-generated-thrift />
+ </target>
+
+
+ <!-- if there's any jni, compile it -->
+
+ <target name="check-jni-source" if="build.jni">
+ <!-- tricksy. remove any target files if any non-target files are newer. -->
+ <dependset>
+ <srcfileset dir="src/main/jni" excludes="**/target/*.${os.jni.ext} **/target/*.jar" />
+ <targetfileset dir="src/main/jni" includes="**/target/*.${os.jni.ext} **/target/*.jar" />
+ </dependset>
+ <!-- and then only set build.jni.ok if those targets are "older" than build.xml (ie missing) -->
+ <uptodate property="build.jni.ok" targetfile="src/main/jni/build.xml">
+ <srcfiles dir="src/main/jni" includes="**/target/*.${os.jni.ext} **/target/*.jar" />
+ </uptodate>
+ </target>
+
+ <target name="compile-jni" depends="check-jni-source" if="build.jni.ok">
+ <ant dir="src/main/jni" target="compile" inheritAll="false" />
+ </target>
+
+ <target name="install-jni" depends="compile-jni" if="build.jni">
+ <copy todir="${dist.dir}" flatten="true">
+ <fileset dir="src/main/jni" includes="**/target/*.${os.jni.ext}" />
+ <fileset dir="src/main/jni" includes="**/target/*.jar" />
+ </copy>
+ </target>
+
+ <!-- compile old-skool java -->
+
+ <target name="compile-java" if="build.java">
+ <javac srcdir="${source.dir}/java" destdir="${target.dir}/classes" deprecation="on">
+ <classpath>
+ <path refid="deps.path" />
+ </classpath>
+ <include name="**/*.java" />
+ </javac>
+ </target>
+
+
+ <!-- compile scala -->
+
+ <target name="compile-scala" if="build.scala">
+ <scalac srcdir="${source.dir}/scala" destdir="${target.dir}/classes" force="changed" deprecation="on" unchecked="on">
+ <classpath>
+ <path refid="deps.path" />
+ </classpath>
+ <include name="**/*.scala" />
+ </scalac>
+ </target>
+
+
+ <!-- create properties file with build info -->
+
+ <condition property="use.git" value="yes">
+ <available file=".git" />
+ </condition>
+
+ <condition property="use.hg" value="yes">
+ <available file=".hg" />
+ </condition>
+
+ <condition property="use.svn" value="yes">
+ <available file=".svn" />
+ </condition>
+
+ <target name="find-git-revision" if="use.git">
+ <!-- ask git for the current "head" commit-id, for memoizing inside the built jar -->
+ <exec outputproperty="revision" executable="git" failifexecutionfails="false">
+ <arg value="rev-parse" />
+ <arg value="HEAD" />
+ </exec>
+ <exec executable="git" failifexecutionfails="false">
+ <arg value="rev-parse" />
+ <arg value="HEAD" />
+ <redirector outputproperty="revision-short">
+ <outputfilterchain>
+ <tokenfilter>
+ <filetokenizer />
+ <replaceregex pattern="(.{8}).*" replace="\1"/>
+ </tokenfilter>
+ </outputfilterchain>
+ </redirector>
+ </exec>
+ </target>
+
+ <target name="find-hg-revision" if="use.hg">
+ <!-- ask hg for the current "tip" commit-id, for memoizing inside the built jar -->
+ <exec outputproperty="revision" executable="hg" failifexecutionfails="false">
+ <arg value="tip" />
+ <arg value="--template" />
+ <arg value="{node}" />
+ </exec>
+ <exec executable="hg" failifexecutionfails="false">
+ <arg value="tip" />
+ <arg value="--template" />
+ <arg value="{node}" />
+ <redirector outputproperty="revision-short">
+ <outputfilterchain>
+ <tokenfilter>
+ <filetokenizer />
+ <replaceregex pattern="(.{8}).*" replace="\1"/>
+ </tokenfilter>
+ </outputfilterchain>
+ </redirector>
+ </exec>
+ </target>
+
+ <target name="find-svn-revision" if="use.svn">
+ <!-- ask svnversion for the revisions in the working copy, for memoizing inside the built jar -->
+ <exec outputproperty="revision" executable="svnversion" failifexecutionfails="false"/>
+ <exec outputproperty="revision-short" executable="svnversion" failifexecutionfails="false"/>
+ </target>
+
+ <target name="find-revision" depends="find-git-revision, find-hg-revision, find-svn-revision" />
+
+ <target name="write-build-info" depends="init, find-revision" if="ivy.extra.buildpackage">
+ <tstamp>
+ <format property="build.timestamp.time" pattern="yyyyMMdd-HHmmss" />
+ <format property="build.timestamp.date" pattern="yyyyMMdd" />
+ </tstamp>
+ <pathconvert property="build.properties.path">
+ <path location="${ivy.extra.buildpackage}" />
+ <unpackagemapper from="${basedir}/*" to="${target.dir}/classes/*" />
+ </pathconvert>
+ <propertyfile file="${build.properties.path}/build.properties">
+ <entry key="name" value="${ivy.module}" />
+ <entry key="version" value="${ivy.revision}" />
+ <entry key="build_name" value="${build.timestamp.time}" />
+ <entry key="build_revision" value="${revision}" />
+ </propertyfile>
+ </target>
+
+
+ <!-- copy resources needed by tests and jar -->
+
+ <target name="copy-resources">
+ <copy todir="${dist.dir}/libs" flatten="true">
+ <path refid="deps.path" />
+ </copy>
+ <copy todir="${target.dir}/test-classes/" failonerror="false">
+ <fileset dir="${test.source.dir}/resources" />
+ </copy>
+ <copy todir="${target.dir}/classes/" overwrite="true" failonerror="false">
+ <fileset dir="${source.dir}/resources" />
+ </copy>
+ </target>
+
+ <target name="copy-config" if="copy.config">
+ <copy todir="${dist.dir}/config">
+ <fileset dir="${basedir}/config" />
+ </copy>
+ </target>
+
+ <target name="copy-extra-config" if="config.extra">
+ <copy todir="${dist.dir}/config">
+ <fileset dir="${config.extra}" />
+ </copy>
+ </target>
+
+ <target name="copy-extra-libs" if="libs.extra">
+ <copy todir="${dist.dir}/libs">
+ <path refid="libs.extra" />
+ </copy>
+ </target>
+
+ <target name="copy-extra-dist" if="dist.extra">
+ <copy todir="${dist.dir}">
+ <path refid="dist.extra" />
+ </copy>
+ </target>
+
+ <target name="copy-extra" depends="copy-resources, copy-config, copy-extra-config, copy-extra-libs, copy-extra-dist" />
+
+
+ <target name="compile" depends="prepare, find-source, compile-thrift, check-jni-source, compile-jni,
+ install-jni, compile-java, compile-scala, write-build-info, copy-extra"
+ description="compile java and scala code" />
+
+</project>
35 ant/docs.xml
@@ -0,0 +1,35 @@
+<project xmlns:ivy="antlib:org.apache.ivy.ant">
+
+ <target name="vscaladoc" depends="prepare" unless="skip.docs">
+ <delete dir="${docs.target.dir}/scaladoc" />
+ <mkdir dir="${docs.target.dir}/scaladoc" />
+ <pathconvert property="doc.sources" pathsep=" ">
+ <fileset dir="${source.dir}" includes="**/*.scala" />
+ </pathconvert>
+ <path id="docs.path">
+ <path refid="bootstrap.path" />
+ <pathelement location="${target.dir}/classes" />
+ </path>
+ <echo message="Building vscaladoc..." />
+ <java classname="org.scala_tools.vscaladoc.Main" fork="true" failonerror="true">
+ <classpath>
+ <path refid="bootstrap.path" />
+ </classpath>
+ <arg value="-classpath" />
+ <arg pathref="docs.path" />
+ <arg value="-d" />
+ <arg value="${docs.target.dir}/scaladoc" />
+ <arg value="-sourcepath" />
+ <arg value="${source.dir}/scala" />
+ <arg value="-windowtitle" />
+ <arg value="${ivy.module} ${ivy.revision}" />
+ <arg value="-doctitle" />
+ <arg value="${ivy.module} ${ivy.revision}" />
+ <arg value="-linksource" />
+ <arg line="${doc.sources}" />
+ </java>
+ </target>
+
+ <target name="docs" depends="prepare,vscaladoc" unless="skip.docs" description="build source documentation" />
+
+</project>
129 ant/package.xml
@@ -0,0 +1,129 @@
+<project xmlns:ivy="antlib:org.apache.ivy.ant">
+
+ <!-- unzip all the dependent jars into target/ so that the final jar will depend on nothing. kinda evil. -->
+ <target name="pack-deps" if="pack.deps">
+ <pathconvert pathsep="," property="deps.path.list">
+ <path refid="deps.path" />
+ <map from="/" to="" />
+ </pathconvert>
+ <unzip dest="${target.dir}/classes">
+ <fileset dir="/" includes="${deps.path.list}" />
+ </unzip>
+ <delete dir="${target.dir}/classes/META-INF" />
+ </target>
+
+ <target name="make-non-executable-jar" unless="ivy.extra.jarclassname" depends="pack-deps">
+ <jar destfile="${dist.dir}/${jar.name}.jar">
+ <fileset dir="${target.dir}/classes" />
+ </jar>
+ </target>
+
+ <!-- generate a jar that contains all deps inside it, so it can be run with "java -jar" -->
+ <target name="make-executable-jar" if="ivy.extra.jarclassname" depends="copy-extra,pack-deps">
+ <pathconvert refid="deps.path" pathsep=" " property="deps.path.jar-format">
+ <chainedmapper>
+ <flattenmapper />
+ <globmapper from="*" to="libs/*" />
+ </chainedmapper>
+ </pathconvert>
+ <jar destfile="${dist.dir}/${jar.name}.jar">
+ <fileset dir="${target.dir}/classes" />
+ <manifest>
+ <attribute name="Main-Class" value="${ivy.extra.jarclassname}" />
+ <attribute name="Class-Path" value="${deps.path.jar-format}" />
+ </manifest>
+ </jar>
+ </target>
+
+ <target name="generate-scripts" depends="prepare" if="generate.scripts">
+ <pathconvert refid="deps.path" property="classpath" />
+ <pathconvert refid="test.path" property="test.classpath" />
+ <pathconvert refid="deps.path" property="deps.path.dist-format">
+ <chainedmapper>
+ <flattenmapper />
+ <globmapper from="*" to="$${DIST_HOME}/libs/*" />
+ </chainedmapper>
+ </pathconvert>
+
+ <!-- delete dir="${basedir}/target/scripts" /-->
+ <mkdir dir="${dist.dir}/scripts" />
+ <copy todir="${dist.dir}/scripts" overwrite="true">
+ <fileset dir="${basedir}/src/scripts" />
+ <filterset>
+ <filter token="CLASSPATH" value="${classpath}:${target.dir}/classes" />
+ <filter token="TEST_CLASSPATH" value="${test.classpath}:${target.dir}/classes:${target.dir}/test-classes" />
+ <filter token="DIST_CLASSPATH" value="${deps.path.dist-format}:$${DIST_HOME}/${jar.name}.jar" />
+ <filter token="TARGET" value="${target.dir}" />
+ <filter token="DIST_NAME" value="${dist.name}" />
+ </filterset>
+ </copy>
+ <copy todir="${dist.dir}/scripts" overwrite="true" failonerror="false">
+ <fileset dir="${target.dir}/gen-rb" />
+ </copy>
+ <chmod dir="${dist.dir}/scripts" includes="*" perm="ugo+x" />
+ </target>
+
+ <target name="jars" depends="test, make-non-executable-jar, make-executable-jar, generate-scripts"
+ description="build jar(s)">
+ <ivy:makepom ivyfile="${basedir}/ivy/ivy.xml" pomfile="${dist.dir}/${jar.name}.pom">
+ <mapping conf="*" scope="compile" />
+ <mapping conf="bootstrap" scope="provided" />
+ <mapping conf="test" scope="provided" />
+ </ivy:makepom>
+ <ivy:deliver conf="*(public)" />
+ </target>
+
+ <target name="package" depends="jars, docs" description="build jar(s) and docs, and package into a tarball">
+ <!-- set revision-short if nothing else has. -->
+ <property name="revision-short" value="HEAD" />
+ <condition property="dist.tarball.name"
+ value="${dist.name}-${revision-short}.tar.bz2"
+ else="${dist.name}.tar.bz2">
+ <isset property="dist.build_integration" />
+ </condition>
+ <tar destfile="${basedir}/dist/${dist.tarball.name}"
+ basedir="${basedir}/dist" compression="bzip2">
+ <include name="${dist.name}/${jar.name}.jar" />
+ <include name="${dist.name}/*.so" />
+ <include name="${dist.name}/*.jnilib" />
+ <include name="${dist.name}/libs/**" />
+ <include name="${dist.name}/config/**" />
+ <tarfileset dir="${basedir}/dist" mode="755">
+ <include name="${dist.name}/scripts/**" />
+ </tarfileset>
+ </tar>
+ <ivy:publish resolver="local" overwrite="true">
+ <artifacts pattern="${dist.dir}/[artifact]-[revision].[ext]" />
+ </ivy:publish>
+ </target>
+
+ <target name="dist" depends="package" description="create a source distribution zipfile">
+ <copy todir="${dist.dir}" failonerror="false">
+ <fileset dir="${basedir}" includes="INSTALL* LICENSE* README* build.xml" />
+ </copy>
+ <copy todir="${dist.dir}/src">
+ <fileset dir="src" />
+ </copy>
+ <copy todir="${dist.dir}/ivy">
+ <fileset dir="ivy" />
+ </copy>
+ <copy todir="${dist.dir}/ant">
+ <fileset dir="ant" />
+ </copy>
+ <zip destfile="${basedir}/dist/${ivy.module}-${ivy.revision}-src.zip"
+ basedir="${basedir}/dist"
+ includes="${ivy.module}-${ivy.revision}/**"
+ excludes="${ivy.module}-${ivy.revision}/libs/** ${ivy.module}-${ivy.revision}/*.jar" />
+ </target>
+
+ <target name="push" depends="package" description="push built jar(s) to the repository">
+ <condition property="pubrevision"
+ value="${ivy.revision}-${build.timestamp.date}-${revision-short}"
+ else="${ivy.revision}">
+ <isset property="push.build_name" />
+ </condition>
+ <ivy:publish resolver="push" overwrite="true" pubrevision="${pubrevision}" update="true">
+ <artifacts pattern="${dist.dir}/[artifact]-${ivy.revision}.[ext]" />
+ </ivy:publish>
+ </target>
+</project>
55 ant/prepare.xml
@@ -0,0 +1,55 @@
+<project xmlns:ivy="antlib:org.apache.ivy.ant">
+
+ <!-- read the ivy files but don't resolve anything yet -->
+ <target name="init" depends="install-ivy">
+ <!-- override some ivy config. must happen before ivy:info even tho it's slightly recursive. -->
+ <property name="ivy.distrib.dir" value="${dist.dir}" />
+ <property name="ivy.deliver.ivy.pattern" value="${dist.dir}/[artifact]-[revision].[ext]" />
+
+ <ivy:info file="${ivy.dep.file}" />
+
+ <condition property="dist.name" value="${ivy.module}" else="${ivy.module}-${ivy.revision}">
+ <isset property="dist.build_integration" />
+ </condition>
+ <property name="jar.name" value="${ivy.module}-${ivy.revision}" />
+ <property name="dist.relative.dir" value="dist/${dist.name}" />
+ <property name="dist.dir" value="${basedir}/${dist.relative.dir}" />
+ <property name="docs.target.dir" value="${dist.dir}/docs" />
+ </target>
+
+ <target name="resolve" depends="init">
+ <ivy:retrieve log="download-only" conf="bootstrap,default,test" pattern="${target.dir}/libs/[conf]/[artifact]-[type].[ext]" />
+ <ivy:cachepath pathid="bootstrap.path" conf="bootstrap" />
+ <ivy:cachepath pathid="deps.path" conf="default" />
+ <ivy:cachepath pathid="test.path" conf="test" />
+ </target>
+
+ <target name="find-source">
+ <pathconvert property="build.java" setonempty="false">
+ <fileset dir="${source.dir}" includes="java/**/*.java" />
+ </pathconvert>
+ <pathconvert property="build.scala" setonempty="false">
+ <fileset dir="${source.dir}" includes="scala/**/*.scala" />
+ </pathconvert>
+ <pathconvert property="build.thrift" setonempty="false">
+ <fileset dir="${source.dir}" includes="thrift/**/*.thrift" />
+ </pathconvert>
+ <pathconvert property="build.jni" setonempty="false">
+ <fileset dir="${source.dir}" includes="jni/build.xml" />
+ </pathconvert>
+ <pathconvert property="copy.config" setonempty="false">
+ <fileset dir="${basedir}" includes="config/**" />
+ </pathconvert>
+ </target>
+
+ <target name="prepare" depends="resolve, find-source" description="bootstrap ivy and scala, and download dependencies">
+ <taskdef resource="scala/tools/ant/antlib.xml" classpathref="bootstrap.path" />
+ <condition property="generate.scripts" value="true">
+ <available file="${basedir}/src/scripts" type="dir" />
+ </condition>
+ <mkdir dir="${target.dir}/classes" />
+ <mkdir dir="${target.dir}/test-classes" />
+ <mkdir dir="${dist.dir}" />
+ </target>
+
+</project>
56 ant/test.xml
@@ -0,0 +1,56 @@
+<project xmlns:ivy="antlib:org.apache.ivy.ant">
+
+ <target name="compile-tests" depends="prepare, compile" unless="skip.test" if="ivy.extra.testclass">
+ <scalac srcdir="${test.source.dir}/scala" destdir="${target.dir}/test-classes" force="changed" deprecation="on">
+ <classpath>
+ <path refid="test.path" />
+ <pathelement location="${target.dir}/classes" />
+ </classpath>
+ <include name="**/*.scala" />
+ </scalac>
+ </target>
+
+ <macrodef name="run-test-class">
+ <attribute name="classname" />
+ <sequential>
+ <java classname="scala.tools.nsc.MainGenericRunner" fork="true" failonerror="true">
+ <classpath>
+ <path refid="test.path" />
+ <pathelement location="${target.dir}/classes" />
+ <pathelement location="${target.dir}/test-classes" />
+ </classpath>
+ <syspropertyset>
+ <propertyref builtin="all" />
+ </syspropertyset>
+ <jvmarg value="-Ddist.dir=${dist.dir}" />
+ <jvmarg value="-Djava.library.path=${dist.dir}:${dist.dir}/libs" />
+ <jvmarg value="-Xmx1200m" />
+ <arg value="@{classname}" />
+ </java>
+ </sequential>
+ </macrodef>
+
+ <target name="run-unit-tests" unless="skip.test" if="ivy.extra.testclass">
+ <run-test-class classname="${ivy.extra.testclass}" />
+ </target>
+
+ <target name="run-stress-tests" unless="skip.test" if="ivy.extra.stresstestclass">
+ <run-test-class classname="${ivy.extra.stresstestclass}" />
+ </target>
+
+ <target name="run-check-tests" unless="skip.test" if="ivy.extra.checktestclass">
+ <run-test-class classname="${ivy.extra.checktestclass}" />
+ </target>
+
+ <target name="unit" depends="compile-tests, run-unit-tests"
+ description="compile and run unit tests" />
+
+ <target name="stress" depends="compile-tests, run-stress-tests"
+ description="compile and run stress tests" />
+
+ <target name="check" depends="compile-tests, run-check-tests"
+ description="compile and run check tests" />
+
+ <target name="test" depends="unit" />
+
+</project>
19 build.xml
@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- CONFIG: change example and description to fit your project -->
+<project name="rowz" default="package" xmlns:ivy="antlib:org.apache.ivy.ant">
+ <description>Simple Row Store using Gizzard</description>
+
+ <!-- use the git revision for the tarball name, instead of the version number: -->
+ <property name="dist.build_integration" value="true" />
+
+ <property name="ivy.settings.file" value="${basedir}/ivy/ivysettings.xml" />
+ <import file="${basedir}/ant/bootstrap.xml" />
+
+ <property name="install.dir" location="/usr/local/rowz" />
+ <target name="install" depends="package">
+ <mkdir dir="${install.dir}" />
+ <copy todir="${install.dir}">
+ <fileset dir="${dist.dir}" includes="**" />
+ </copy>
+ </target>
+</project>
6 config/development.conf
@@ -0,0 +1,6 @@
+log {
+ level = "info"
+ console = false
+ filename = "rowz.log"
+ roll = "never"
+}
5 config/production.conf
@@ -0,0 +1,5 @@
+log {
+ filename = "/var/log/rowz/production.log"
+ level = "info"
+ roll = "hourly"
+}
4 config/test.conf
@@ -0,0 +1,4 @@
+log {
+ level = "fatal"
+ console = true
+}
42 ivy/ivy.xml
@@ -0,0 +1,42 @@
+<ivy-module version="1.0" xmlns:e="http://ant.apache.org/ivy/extra">
+ <info organisation="com.twitter"
+ module="rowz"
+ revision="1.0"
+ e:buildpackage="com.twitter.rowz"
+ e:testclass="com.twitter.rowz.TestRunner"
+ e:jarclassname="com.twitter.rowz.Main"
+ e:thriftpackage="com.twitter.rowz.gen"
+ />
+
+ <configurations>
+ <conf name="bootstrap" visibility="private" description="load scala compiler and libraries" />
+ <conf name="default" description="normal build" />
+ <conf name="test" extends="default" visibility="private" description="build and run tests" />
+ </configurations>
+
+ <publications>
+ <artifact />
+ <artifact type="pom" />
+ </publications>
+
+ <dependencies>
+ <dependency org="org.scala-lang" name="scala-compiler" rev="2.7.7" />
+ <dependency org="org.scala-lang" name="scala-library" rev="2.7.7" />
+ <dependency org="org.scala-tools.testing" name="specs" rev="1.6.1" conf="test->*" />
+ <dependency org="org.scala-tools" name="vscaladoc" rev="1.1-md-3" conf="bootstrap->*" />
+
+ <dependency org="net.lag" name="configgy" rev="1.4" />
+ <dependency org="commons-logging" name="commons-logging" rev="1.1" />
+ <dependency org="commons-lang" name="commons-lang" rev="2.2" />
+ <dependency org="com.twitter" name="ostrich" rev="1.0" conf="*" /> <!--auto-->
+ <dependency org="com.twitter" name="gizzard" rev="1.0" conf="*" /> <!--auto-->
+ <dependency org="com.twitter" name="querulous" rev="1.1.4" />
+ <dependency org="thrift" name="libthrift" rev="751142" conf="*" /> <!--auto-->
+ <dependency org="org.jmock" name="jmock" rev="2.4.0" conf="test->*" /> <!--auto-->
+ <dependency org="org.hamcrest" name="hamcrest-all" rev="1.1" conf="test->*" /> <!--auto-->
+ <dependency org="cglib" name="cglib" rev="2.1_3" conf="test->*" /> <!--auto-->
+ <dependency org="asm" name="asm" rev="1.5.3" conf="test->*" /> <!--auto-->
+ <dependency org="org.objenesis" name="objenesis" rev="1.1" conf="test->*" /> <!--auto-->
+ <dependency org="com.twitter" name="xrayspecs" rev="1.0.7" conf="*" /> <!--auto-->
+ </dependencies>
+</ivy-module>
30 ivy/ivysettings.xml
@@ -0,0 +1,30 @@
+<ivysettings>
+ <settings defaultResolver="chain-repos" />
+ <resolvers>
+ <chain name="chain-repos" returnFirst="true">
+ <filesystem name="local-libs" m2compatible="true" transactional="false" local="true" checkmodified="true">
+ <artifact pattern="${basedir}/libs/[artifact]-[revision].[ext]" />
+ </filesystem>
+
+ <ibiblio name="scala-tools.org" m2compatible="true" root="http://scala-tools.org/repo-releases/" />
+ <ibiblio name="specs-extra" m2compatible="true" root="http://specs.googlecode.com/svn/maven2/" />
+ <ibiblio name="maven2-repository.dev.java.net" m2compatible="true" root="http://download.java.net/maven/2/"/>
+ <!-- for smack, smackx -->
+ <ibiblio name="reucon" m2compatible="true" root="http://maven.reucon.com/public/" />
+ <ibiblio name="maven2" m2compatible="true" usepoms="false" />
+ <ibiblio name="lag.net" m2compatible="true" root="http://www.lag.net/repo/" />
+ <ibiblio name="twitter.com" m2compatible="true" root="http://www.lag.net/nest/" />
+ <!-- for oauth -->
+ <ibiblio name="oauth.net" m2compatible="true" root="http://oauth.googlecode.com/svn/code/maven" />
+ <ibiblio name="powermock" m2compatible="true" root="http://powermock.googlecode.com/svn/repo/" />
+ <ibiblio name="jboss" m2compatible="true" root="http://repository.jboss.org/maven2/" />
+ <ibiblio name="mirrors.ibiblio.org" m2compatible="true" root="http://mirrors.ibiblio.org/pub/mirrors/maven2/" />
+ <ibiblio name="atlassian" m2compatible="true" root="https://m2proxy.atlassian.com/repository/public/" />
+ </chain>
+ <filesystem name="local" m2compatible="true" transactional="false">
+ <artifact pattern="${basedir}/dist/repo/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
+ </filesystem>
+ </resolvers>
+
+ <caches useOrigin="true" />
+</ivysettings>
BIN  libs/gizzard-1.0.jar
Binary file not shown
9 src/main/scala/com/twitter/rowz/CopyManager.scala
@@ -0,0 +1,9 @@
+package com.twitter.rowz
+
+import com.twitter.gizzard.nameserver
+import com.twitter.gizzard.scheduler.JobScheduler
+
+
+class CopyManager(val scheduler: JobScheduler) extends nameserver.CopyManager[Shard] {
+
+}
10 src/main/scala/com/twitter/rowz/ForwardingManager.scala
@@ -0,0 +1,10 @@
+package com.twitter.rowz
+
+import com.twitter.gizzard.nameserver
+import com.twitter.querulous.evaluator.QueryEvaluator
+
+
+class ForwardingManager(mappingFunction: Long => Long, protected val queryEvaluator: QueryEvaluator)
+ extends nameserver.ForwardingManager[Shard] {
+
+}
20 src/main/scala/com/twitter/rowz/Hash.scala
@@ -0,0 +1,20 @@
+package com.twitter.rowz
+
+import java.security.MessageDigest
+import java.nio.{ByteBuffer, ByteOrder}
+
+
+object Hash extends (Long => Long) {
+ def apply(n: Long) = {
+ val buffer = new Array[Byte](8)
+ val byteBuffer = ByteBuffer.wrap(buffer)
+ byteBuffer.order(ByteOrder.LITTLE_ENDIAN)
+ byteBuffer.putLong(n)
+ val digest = MessageDigest.getInstance("SHA")
+ digest.update(buffer)
+ val longs = ByteBuffer.wrap(digest.digest).asLongBuffer
+ val results = new Array[Long](longs.limit)
+ longs.get(results)
+ results.first
+ }
+}
68 src/main/scala/com/twitter/rowz/Main.scala
@@ -0,0 +1,68 @@
+package com.twitter.rowz
+
+import net.lag.configgy.Configgy
+import net.lag.logging.Logger
+import com.twitter.gizzard.nameserver.NameServer
+import com.twitter.gizzard.scheduler.JobScheduler
+import com.twitter.gizzard.thrift.{TSelectorServer, JobManagerService}
+import com.facebook.thrift.server.{TServer, TThreadPoolServer}
+import com.facebook.thrift.transport.{TServerSocket, TTransportFactory}
+import com.twitter.ostrich.W3CStats
+
+
+object Main {
+ var rowzService: RowzService = null
+ var nameServer: NameServer[Shard] = null
+ var scheduler: JobScheduler = null
+ var rowzServer: TSelectorServer = null
+ var jobServer: TSelectorServer = null
+ var shardServer: TSelectorServer = null
+
+ val config = Configgy.config
+ val w3c = new W3CStats(Logger.get("w3c"), Array(
+ "action-timing",
+ "db-timing",
+ "connection-pool-release-timing",
+ "connection-pool-reserve-timing",
+ "kestrel-put-timing",
+ "db-select-count",
+ "db-execute-count",
+ "operation",
+ "arguments"
+ ))
+
+ def main(args: Array[String]) {
+ val state = Rowz(config)
+ rowzService = state._0
+ nameServer = state._1
+ scheduler = state._2
+
+ startThrift()
+ }
+
+ def startThrift() {
+ val executor = TSelectorServer.makeThreadPoolExecutor(config)
+ val processor = new rowz.thrift.Rowz.Processor(ExceptionWrappingProxy[rowz.thrift.Rowz.Iface](LoggingProxy[rowz.thrift.Rowz.Iface](Stats, w3c, "Rowz", rowzService)))
+ rowzServer = TSelectorServer("rowz", config("port").toInt, processor, executor, config("timeout").toInt.milliseconds)
+
+ val jobService = new JobManagerService(scheduler)
+ val jobProcessor = new JobManager.Processor(ExceptionWrappingProxy[JobManager.Iface](LoggingProxy[JobManager.Iface](Stats, Main.w3c, "RowzJobs", jobService)))
+ jobServer = TSelectorServer("rowz-jobs", config("edges.job_server_port").toInt, jobProcessor, executor, config("timeout").toInt.milliseconds)
+
+ val shardService = new ShardManagerService(nameServer)
+ val shardProcessor = new ShardManager.Processor(ExceptionWrappingProxy[ShardManager.Iface](LoggingProxy[ShardManager.Iface](Stats, Main.w3c, "RowzShards", shardService)))
+ shardServer = TSelectorServer("edges-shards", config("edges.shard_server_port").toInt, edgesShardProcessor, edgesExecutor, edgesTimeout)
+
+ rowzServer.serve()
+ jobServer.serve()
+ shardServer.serve()
+ }
+
+ def shutdown() {
+ rowzServer.stop()
+ jobServer.stop()
+ scheduler.shutdown()
+
+ System.exit(0)
+ }
+}
57 src/main/scala/com/twitter/rowz/Rowz.scala
@@ -0,0 +1,57 @@
+package com.twitter.rowz
+
+import net.lag.configgy.Config
+import com.twitter.querulous.database.{ApachePoolingDatabaseFactory, MemoizingDatabaseFactory}
+import com.twitter.querulous.query.SqlQueryFactory
+import com.twitter.querulous.evaluator.StandardQueryEvaluatorFactory
+import com.twitter.xrayspecs.TimeConversions._
+import net.lag.logging.{Logger, ThrottledLogger}
+import com.twitter.gizzard.Future
+import com.twitter.gizzard.scheduler.JobScheduler
+import com.twitter.gizzard.nameserver.{NameServer, ShardRepository}
+import com.twitter.gizzard.jobs.PolymorphicJobParser
+
+
+object Rowz {
+ def apply(config: Config) = {
+ val databaseFactory = new MemoizingDatabaseFactory(new ApachePoolingDatabaseFactory(
+ config("rowz.db.connection_pool.size_min").toInt,
+ config("rowz.db.connection_pool.size_max").toInt,
+ config("rowz.db.connection_pool.test_idle_msec").toLong.millis,
+ config("rowz.db.connection_pool.max_wait").toLong.millis,
+ config("rowz.db.connection_pool.test_on_borrow").toBoolean,
+ config("rowz.db.connection_pool.min_evictable_idle_msec").toLong.millis))
+
+ val queryEvaluatorFactory = new StandardQueryEvaluatorFactory(databaseFactory, new SqlQueryFactory)
+ val nameServerQueryEvaluator = queryEvaluatorFactory(
+ config("nameserver.hostname"),
+ config("nameserver.database"),
+ config("nameserver.username"),
+ config("nameserver.password"))
+
+ val throttledLogger = new ThrottledLogger[String](Logger(), config("throttled_log.period_msec").toInt, config("throttled_log.rate").toInt)
+ val future = new Future("ReplicatingFuture", config.configMap("replication.future"))
+
+ val shardRepository = new ShardRepository[Shard]
+ shardRepository += ("com.twitter.rowz.SqlShard" -> new SqlShardFactory(queryEvaluatorFactory, config))
+ shardRepository += ("com.twitter.gizzard.ReadOnlyShard" -> new gizzard.ReadOnlyShardFactory)
+ shardRepository += ("com.twitter.gizzard.BlockedShard" -> new gizzard.BlockedShardFactory)
+ shardRepository += ("com.twitter.gizzard.WriteOnlyShard" -> new gizzard.WriteOnlyShardFactory)
+ shardRepository += ("com.twitter.gizzard.ReplicatingShard" -> new gizzard.ReplicatingShardFactory(throttledLogger, replicatingFuture))
+
+ val polymorphicJobParser = new PolymorphicJobParser
+ val scheduler = JobScheduler("jobs", queueConfig, jobParser, Main.w3c)
+
+ val queryExecutorFuture = new Future("QueryExecutorFuture", config.configMap("groups.query_executor_future"))
+
+ val forwardingManager = new ForwardingManager(Hash, nameServerQueryEvaluator)
+ val copyManager = new CopyManager(scheduler)
+ val nameServer = new NameServer[Shard](nameServerQueryEvaluator, shardRepository, forwardingManager, copyManager)
+ val rowzService = new RowzService(nameServer, forwardingManager, scheduler)
+
+ nameServer.reload()
+ scheduler.start()
+
+ (rowzService, nameServer, scheduler)
+ }
+}
6 src/main/scala/com/twitter/rowz/Shard.scala
@@ -0,0 +1,6 @@
+package com.twitter.rowz
+
+
+trait Shard {
+
+}
6 src/main/scala/com/twitter/rowz/jobs/Create.scala
@@ -0,0 +1,6 @@
+package com.twitter.rowz.jobs
+
+
+class Create {
+
+}
6 src/main/scala/com/twitter/rowz/jobs/Delete.scala
@@ -0,0 +1,6 @@
+package com.twitter.rowz.jobs
+
+
+class Delete {
+
+}
22 src/main/scala/com/twitter/rowz/thrift/RowzService.scala
@@ -0,0 +1,22 @@
+package com.twitter.rowz
+
+import net.lag.configgy.Config
+import com.twitter.gizzard.nameserver.NameServer
+import com.twitter.gizzard.scheduler.JobScheduler
+
+
+class RowzService(nameServer: NameServer[Shard], forwardingManager: ForwardingManager, scheduler: JobScheduler) {
+ def create(rowInfo: RowInfo, at: Int) = {
+ val id = makeId()
+ scheduler(new Create(id, rowInfo.fromThrift, at))
+ id
+ }
+
+ def delete(rowInfo: RowInfo, at: Int) {
+ scheduler(new Delete(id, rowInfo.fromThrift, at))
+ }
+
+ def get(id: Long) = {
+ nameServer.find(id).get(id)
+ }
+}
22 src/main/thrift/Rowz.thrift
@@ -0,0 +1,22 @@
+namespace java com.twitter.rowz.thrift
+namespace rb Rowz
+
+struct RowInfo {
+ 1: string name
+ 2: i32 state_id
+}
+
+struct Row {
+ 1: i64 id
+ 2: RowInfo info
+}
+
+exception RowzException {
+ 1: string description
+}
+
+service Groups {
+ i64 create(1: RowInfo info, 2: i32 at) throws(RowzException ex)
+ void destroy(1: i64 id, 2: i32 at) throws(RowzException ex)
+ RowInfo read(1: i64 id) throws(RowzException ex)
+}
127 src/scripts/rowz.sh
@@ -0,0 +1,127 @@
+#!/bin/sh
+#
+# rowz init.d script.
+#
+# All java services require the same directory structure:
+# /opt/local/$APP_NAME-$VERSION
+# /var/log/$APP_NAME (chown daemon, chmod 775)
+
+APP_NAME="rowz"
+VERSION="1.0"
+APP_HOME="/opt/local/$APP_NAME/current"
+AS_USER="daemon"
+DAEMON="/usr/local/bin/daemon"
+
+HEAP_OPTS="-Xmx13000m -Xms13000m -XX:NewSize=1024m"
+JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
+GC_OPTS="-verbosegc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:+UseParNewGC"
+GC_LOG="-Xloggc:/var/log/$APP_NAME/gc.log"
+DEBUG_OPTS="-XX:ErrorFile=/var/log/$APP_NAME/java_error%p.log"
+JAVA_OPTS="-server $GC_OPTS $GC_LOG $HEAP_OPTS $JMX_OPTS $DEBUG_OPTS"
+
+pidfile="/var/run/$APP_NAME/$APP_NAME.pid"
+daemon_args="--name $APP_NAME --pidfile $pidfile"
+daemon_start_args="--user $AS_USER --stdout=/var/log/$APP_NAME/stdout --stderr=/var/log/$APP_NAME/error"
+
+function running() {
+ $DAEMON $daemon_args --running
+}
+
+function find_java() {
+ if [ ! -z "$JAVA_HOME" ]; then
+ return
+ fi
+ for dir in /opt/jdk /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home /usr/java/default; do
+ if [ -x $dir/bin/java ]; then
+ JAVA_HOME=$dir
+ break
+ fi
+ done
+}
+
+# dirs under /var/run can go away between reboots.
+for p in /var/run/$APP_NAME /var/log/$APP_NAME; do
+ if [ ! -d $p ]; then
+ mkdir -p $p
+ chmod 775 $p
+ chown $AS_USER $p >/dev/null 2>&1 || true
+ fi
+done
+
+find_java
+
+
+case "$1" in
+ start)
+ echo -n "Starting $APP_NAME... "
+
+ if [ ! -r $APP_HOME/$APP_NAME-$VERSION.jar ]; then
+ echo "FAIL"
+ echo "*** $APP_NAME jar missing: $APP_HOME/$APP_NAME-$VERSION.jar - not starting"
+ exit 1
+ fi
+ if [ ! -x $JAVA_HOME/bin/java ]; then
+ echo "FAIL"
+ echo "*** $JAVA_HOME/bin/java doesn't exist -- check JAVA_HOME?"
+ exit 1
+ fi
+ if running; then
+ echo "already running."
+ exit 0
+ fi
+
+ ulimit -n 8192 || echo -n " (no ulimit)"
+ $DAEMON $daemon_args $daemon_start_args -- ${JAVA_HOME}/bin/java ${JAVA_OPTS} -jar ${APP_HOME}/${APP_NAME}-${VERSION}.jar
+ tries=0
+ while ! running; do
+ tries=$((tries + 1))
+ if [ $tries -ge 5 ]; then
+ echo "FAIL"
+ exit 1
+ fi
+ sleep 1
+ done
+ echo "done."
+ ;;
+
+ stop)
+ echo -n "Stopping $APP_NAME... "
+ if ! running; then
+ echo "wasn't running."
+ exit 0
+ fi
+
+ kill -TERM $(cat $pidfile)
+ tries=0
+ while running; do
+ tries=$((tries + 1))
+ if [ $tries -ge 5 ]; then
+ echo "FAIL"
+ exit 1
+ fi
+ sleep 1
+ done
+ echo "done."
+ ;;
+
+ status)
+ if running; then
+ echo "$APP_NAME is running."
+ else
+ echo "$APP_NAME is NOT running."
+ fi
+ ;;
+
+ restart)
+ $0 stop
+ sleep 2
+ $0 start
+ ;;
+
+ *)
+ echo "Usage: /etc/init.d/${APP_NAME}.sh {start|stop|restart|status}"
+ exit 1
+ ;;
+esac
+
+exit 0
12 src/test/scala/com/twitter/rowz/SampleSpec.scala
@@ -0,0 +1,12 @@
+package com.twitter.rowz
+
+import org.specs.Specification
+import org.specs.mock.{ClassMocker, JMocker}
+
+object SampleSpec extends Specification with JMocker with ClassMocker {
+ "Sample" should {
+ "run a test" in {
+ 23 mustEqual 23
+ }
+ }
+}
5 src/test/scala/com/twitter/rowz/TestRunner.scala
@@ -0,0 +1,5 @@
+package com.twitter.rowz
+
+import com.twitter.xrayspecs.XraySpecsRunner
+
+object TestRunner extends XraySpecsRunner
Please sign in to comment.
Something went wrong with that request. Please try again.