Browse files

Merge branch 'master' of git://

* 'master' of git://
  Add optimize command/function and tests, fix Makefile test runner
  Added optimize command to principe
  Minor tweakage
  port server fixes
  doc cleanup
  • Loading branch information...
2 parents 7432095 + 4243448 commit f3a9b1d28db56d41c2783e4abad2e9b5e3343304 @yrashk committed Aug 12, 2009
Showing with 144 additions and 103 deletions.
  1. +5 −5 Makefile
  2. +3 −10 README
  3. +0 −3 TODO
  4. +42 −15 doc/overview.edoc
  5. +23 −59 src/medici_port_srv.erl
  6. +51 −8 src/principe.erl
  7. +10 −3 src/principe_table.erl
  8. +10 −0 test/principe_test.erl
@@ -16,19 +16,19 @@ clean:
rm -fv test/*.beam
# Testing with a Tokyo Tyrant server instance
-test: clean ttclean tt_normal testbuild run_basic_test ttstopd tt_table run_table_test ttstopd ttclean
+test: clean ttclean tt_normal testbuild run_basic_test tt_table run_table_test ttstopd #ttclean
erlc -o ebin/ src/*.erl
- erlc test/*.erl
+ erlc -o test/ test/*.erl
erl -pa ebin/ -pa test/ -noshell -s principe_test test -s init stop
erl -pa ebin/ -pa test/ -noshell -s principe_table_test test -s init stop
rm -f /tmp/ /tmp/ttserver.*
- ttserver -dmn -pid /tmp/ /tmp/ttserver.tcb
+ ttserver -dmn -kl -pid /tmp/ /tmp/ttserver.tch
- ttserver -dmn -pid /tmp/ /tmp/ttserver.tct
+ ttserver -dmn -kl -pid /tmp/ /tmp/ttserver.tct
- kill -TERM `cat /tmp/`
+ kill -TERM `head -1 /tmp/`
@@ -10,16 +10,9 @@ to a tyrant server and returns the results.
Medici is an Erlang app that provides a supervised pool of client
connections to the tyrant service (via principe) and a client controller
that provides an fairly robust api for other processes to use when
-accessing the tyrant service.
-Keys and Values
-All keys for database operations should be iolists. Values can be
-iolists, integers, or floats. For the table interface the columns
-are accessed via {column_name, column_value} proplists, where the
-column name is an iolist and the column value is an iolist, integer,
-or float.
+accessing the tyrant service. The medici app can also run a tyrant
+server as a port, controlling the server from Erlang and eliminating
+the necessity to start/stop this server outside of the Erlang application.
@@ -9,6 +9,3 @@ table search functions useless.
medici: need to flesh-out/test support for running multiple medici
instances (e.g. one for tyrant server A and one for tyrant server B)
-medici: add support for running the tyrant server directly within medici
-(as a port, with tyrant stdio logging going to the erlang logging facilities).
@@ -11,8 +11,10 @@ interfaces for the Tokyo Tyrant network database interface. Principe
provides a thin wrapper for the Tyrant TCP/IP protocol but does not do
any connection management beyond providing a simple connect function.
The medici application provides connection management, a non-blocking
-interface for the pool of principe connections, and standard OTP
-restart/fault-tolerance capabilities.
+interface for the pool of principe connections, standard OTP
+restart/fault-tolerance capabilities, and the ability to run the
+Tyrant server as an Erlang port for full control of the server from
+within Erlang.
== Using Principe ==
@@ -89,26 +91,51 @@ to the Tyrant server so that you don't need to worry about it.
=== Configuration/Connection Options ===
-The various options available for setting configuration or connection
-parameters. In addition to the hostname, port, connect_opts that are
-used by the principe/principe_table modules, the following options are
-used by the
+The various options are available for setting configuration or connection
-controller: an atom that is used as the registered name for the medici
+The principe and principe_table connect() function uses the hostname,
+port, connect_opts to tell gen_tcp where the Tyrant server is located.
+The medici application configuration options can be set as OTP
+application env values. The "options" env key has as its value a
+proplist that contains the various configuration options to be used.
+Modify the file for making persistent changes, tweak the
+values from the command-line for changing the values when the Erlang
+VM starts, or pass in the options proplist as the argument to
+medici:start() to modify the settings at runtime.
+The following options are examined and used by medici to modify how it
+operates, each option is a key in the options proplist with a specific
+value (and meaning for that value) as follows:
-native: a boolean value (false is assumed if property is not present
-in configuration options) that turns the medici service into an erlang
-term storage server. Erlang terms will be used as the keys and all
-values returned will be erlang terms. If the native flag is set in
+controller: If present the value should be an atom that is used as the
+registered name for the medici service. If not present then the
+service is registered using the name 'medici'.
+native: This is a boolean value (false is assumed if property is not
+present in configuration options) that turns the medici service into
+an erlang term storage server. Erlang terms will be used as the keys
+and all values returned will be erlang terms. If the native flag is set in
the configuration options the tyrant server cannot be of the table or
fixed-length types.
-num_connections: an integer that tells the connection controler how
-many connection clients to keep active in the pool. The default of 8
-is also the number of work threads that a tyrant server will start as
+num_connections:This is an integer that tells the connection controler
+how many connection clients to keep active in the pool. The default
+of 8 is also the number of work threads that a tyrant server will start as
its default, so if you change this number you should also start your
tyrant server with an equivalent number of worker threads for best
+run_server: This value is used to tell the medici application to run
+the Tyrant server itself using a port server. The value for this
+property is a proplist that contains up to five additional key/value
+pairs: tyrant_bin, tyrant_opts, data_file, tuning_opts, and
+port_opts. The value of the port_opts proplist element should be a
+list containing valid options for the erlang:open_port() function, the
+remainder are strings containing the pathname for the ttserver binary
+(tyrant_bin), the command-line options for ttserver (tyrant_opts), the
+pathname of the database file to use (data_file), and any additional
+tuning options to apply to this database (tuning_opts).
@@ -1,9 +1,21 @@
+%%%% The contents of this file are subject to the Erlang Public License,
+%%% Version 1.1, (the "License"); you may not use this file except in
+%%% compliance with the License. You should have received a copy of the
+%%% Erlang Public License along with this software. If not, it can be
+%%% retrieved via the world wide web at
+%%% Software distributed under the License is distributed on an "AS IS"
+%%% basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+%%% the License for the specific language governing rights and limitations
+%%% under the License.
-%%% File : medici_port_srv.erl
-%%% Author : Jim McCoy <>
-%%% Description :
+%%% File: medici_port_srv.erl
+%%% @author Jim McCoy <>
+%%% @copyright Copyright (c) 2009, Jim McCoy. All Rights Reserved.
-%%% Created : 30 May 2009 by Jim McCoy <>
+%%% @private
+%%% An Erlang port server that manages a Tyrant server.
+%%% @end
@@ -16,15 +28,6 @@
-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
terminate/2, code_change/3]).
-%% The options part of the state record is a collection of proplists
-%% for the various tokyo tyrant options. There are five key/string proplists
-%% in the tyrant_command list:
-%% server_bin - single element, the path to the tyrant binary
-%% data_file - single element, the path to the database file
-%% tyrant_opts - proplist, key/value list of command-line tyrant flags
-%% tuning_opts - proplist, key/value list of tyrant tuning options (appended
-%% to data_file name when invoking tyrant command)
-%% port_opts - list of options to be used by open_port
-record(state, {port=nil,
@@ -42,25 +45,16 @@
%% API
-%% Function: start_link() -> {ok,Pid} | ignore | {error,Error}
-%% Description: Starts the server
+%% @spec start_link() -> {ok, Pid} | {error, term()}
+%% @private Start the Tyrant port server
start_link() ->
gen_server:start_link(?MODULE, [], []).
%% gen_server callbacks
-%% Function: init(Args) -> {ok, State} |
-%% {ok, State, Timeout} |
-%% ignore |
-%% {stop, Reason}
-%% Description: Initiates the server
-init([]) ->
+init(_Args) ->
{ok, LogMatch} = re:compile(?LOG_REGEXP),
{ok, PidMatch} = re:compile(?PID_REGEXP),
case application:get_env(options) of
@@ -73,15 +67,6 @@ init([]) ->
start_server(ServerOpts, #state{log_match=LogMatch,
-%% Function: %% handle_call(Request, From, State) -> {reply, Reply, State} |
-%% {reply, Reply, State, Timeout} |
-%% {noreply, State} |
-%% {noreply, State, Timeout} |
-%% {stop, Reason, Reply, State} |
-%% {stop, Reason, State}
-%% Description: Handling call messages
handle_call({get_info}, _From, State) ->
{reply, {State#state.options,}, State};
handle_call({restart, ServerOpts}, _From, State) ->
@@ -101,45 +86,24 @@ handle_call({restart}, _From, State) ->
handle_call({stop}, _From, State) ->
{stop, asked_to_stop, State}.
-%% Function: handle_cast(Msg, State) -> {noreply, State} |
-%% {noreply, State, Timeout} |
-%% {stop, Reason, State}
-%% Description: Handling cast messages
handle_cast(_Msg, State) ->
{noreply, State}.
-%% Function: handle_info(Info, State) -> {noreply, State} |
-%% {noreply, State, Timeout} |
-%% {stop, Reason, State}
-%% Description: Handling all non call/cast messages
handle_info({'EXIT', Port, Reason}, #state{port=Port} = State) ->
- {stop, {port_terminated, Reason}, State};
+ {stop, {port_terminated, Reason},State#state{port=nil, pid=0}};
+handle_info({Port, closed}, #state{port=Port} = State) ->
+ {stop, {port_terminated, returned_close_msg},State#state{port=nil, pid=0}};
handle_info({Port, {data, {eol, StdOutMsg}}}, #state{port=Port} = State) ->
parse_log_message(binary_to_list(StdOutMsg), State);
handle_info(Info, State) ->
io:format("unrecognized info message: ~p~n", [Info]),
{noreply, State}.
-%% Function: terminate(Reason, State) -> void()
-%% Description: This function is called by a gen_server when it is about to
-%% terminate. It should be the opposite of Module:init/1 and do any necessary
-%% cleaning up. When it returns, the gen_server terminates with Reason.
-%% The return value is ignored.
terminate({port_terminated, _Reason}, _State) ->
terminate(_Reason, State) ->
-%% Func: code_change(OldVsn, State, Extra) -> {ok, NewState}
-%% Description: Convert process state when code is changed
code_change(_OldVsn, State, _Extra) ->
{ok, State}.
@@ -57,14 +57,16 @@
- sync/1,
+ sync/1,
+ optimize/2,
- restore/3,
- setmst/3,
+ restore/3,
+ restore_with_check/3,
+ setmst/3,
@@ -94,14 +96,16 @@
-define(ADDDOUBLE, 16#C861).
-define(EXT, 16#C868).
-define(SYNC, 16#C870).
--define(VANISH, 16#C871).
--define(COPY, 16#C872).
--define(RESTORE, 16#C873).
+-define(OPTIMIZE, 16#C871).
+-define(VANISH, 16#C872).
+-define(COPY, 16#C873).
+-define(RESTORE, 16#C874).
-define(SETMST, 16#C878).
-define(RNUM, 16#C880).
-define(SIZE, 16#C881).
-define(STAT, 16#C888).
-define(MISC, 16#C890).
+-define(REPL, 16#C8A0).
-define(MONOULOG, 1 bsl 0).
-define(XOLCKREC, 1 bsl 0).
@@ -487,6 +491,16 @@ vanish(Socket) ->
+%% @spec optimize(Socket::port(),
+%% iolist()) -> ok | error()
+%% @doc Change the remote database tuning parameters. The second parameter
+%% should be a list of the database tuning parameters that will be applied
+%% at the remote end (e.g. "#bnum=1000000#opts=ld").
+optimize(Socket, Key) ->
+ ?T1(?OPTIMIZE), % Using 'Key' so that the macro binds properly...
%% @spec rnum(Socket::port()) -> integer() | error()
%% @doc Get the number of records in the remote database.
@@ -532,16 +546,33 @@ copy(Socket, Key) when is_binary(Key) ->
%% @spec restore(Socket::port(),
%% PathName::iolist(),
-%% TimeStamp::integer) -> ok | error()
+%% TimeStamp::integer,
+%% Options::) -> ok | error()
%% @doc Restore the database to a particular point in time from the update log.
restore(Socket, PathName, TimeStamp) ->
gen_tcp:send(Socket, [<<?RESTORE:16>>,
- <<TimeStamp:64>>,
+ <<TimeStamp:64>>,
+ <<0:32>>,
+%% @spec restore_with_check(Socket::port(),
+%% PathName::iolist(),
+%% TimeStamp::integer) -> ok | error()
+%% @doc Restore the database to a particular point in time from the update log and
+%% perform a consistency check
+%% @end
+restore_with_check(Socket, PathName, TimeStamp) ->
+ gen_tcp:send(Socket, [<<?RESTORE:16>>,
+ <<(iolist_size(PathName)):32>>,
+ <<TimeStamp:64>>,
+ <<1:32>>,
+ PathName]),
%% @spec setmst(Socket::port(),
%% HostName::iolist(),
%% Port::integer) -> ok | error()
@@ -553,6 +584,18 @@ setmst(Socket, HostName, Port) when is_integer(Port) ->
<<Port:32>>, HostName]),
+%% @spec repl(Socket::port(),
+%% TimeStamp::integer(),
+%% Sid::integer())
+%% @doc Initiate master->slave replication to the server id provided starting at a
+%% given timestamp.
+%% repl(Socket, TimeStamp, Sid) ->
+%% gen_tcp:send(Socket, [<<?SETMST:16>>,
+%% <<TimeStamp:64>>,
+%% <<Sid:32>>]),
%% @spec misc(Socket::port(),
%% Func::iolist(),
%% Args::arglist()) -> [binary()] | error()
Oops, something went wrong.

0 comments on commit f3a9b1d

Please sign in to comment.