Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Test Hibari deployment on Linux ARMv7 based scale out cluster (Calxeda EnergyCore) #25

Open
tatsuya6502 opened this Issue · 9 comments

1 participant

@tatsuya6502
Owner

Deploy Hibari on a test cluster of Calxeda EnergyCore servers running ARM Ubuntu 12.04 LTS (Linux ARMv7 architecture). Create documentation, run Basho Bench for performance measure and a 48-hour stability test. If possible, work as a project at Cloudian, Inc. to create a white paper for Calxeda's Software Partner page.

  • Create installation instructions for ARM Ubuntu
  • Create sample Hibari configurations
  • Develop and run custom Basho Bench drivers that simulate the workloads of typical applications for Hibari
  • Write a white paper with performance measure that can be published on the web sites of Calxeda, Altima and Cloudian
@tatsuya6502 tatsuya6502 was assigned
@tatsuya6502
Owner

I have arranged a Calxeda based evaluation cluster to be delivered to our Tokyo office in April 2013. I believe Calxeda's ARM based SoC with on-chip high-speed fabric switch must be very good for distributed storage. Calxeda claims their system will yield more IOPs than traditional x86 based system. I'm looking forward to seeing the result.

Calxeda finds a new market in storage - Gigamon
http://gigaom.com/2012/12/27/calxeda-finds-a-new-market-in-storage/

Its tests show roughly a 4X improvement in IOPs for a rack of
Calxeda SoCs versus x86-based systems. Adding Calexeda’s SoCs also
cuts complexity because the entire system of processing and
networking components are integrated on the SoC, and the
terabit-plus fabric between cores also offers more network capacity
between cores in a system -- the so-called east-west networking
traffic.

@tatsuya6502
Owner

I had no problem for compiling Hibari v0.1.10 on Calxeda Greenbox evaluation system running Ubuntu 12.04.2 LTS.

ubuntu@esp040:~/hibari/hibari$ uname -a
Linux esp040.******.com 3.2.0-40-highbank #64-Ubuntu SMP PREEMPT Tue Mar 26 01:14:33 UTC 2013 armv7l armv7l armv7l GNU/Linux

ubuntu@esp040:~/hibari/hibari$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS"

ubuntu@esp040:~/hibari/hibari$ time make package
cleaning: hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32 ...
./rebar clean
==> gmt_util (clean)
==> riak_err (clean)
==> cluster_info (clean)
==> partition_detector (clean)
==> congestion_watcher (clean)
==> meck (clean)
==> edown (clean)
==> asciiedoc (clean)
==> ubf (clean)
==> ubf_thrift (clean)
==> gdss_brick (clean)
==> gdss_client (clean)
==> gdss_admin (clean)
==> gdss_ubf_proto (clean)
==> rel (clean)
==> hibari (clean)
compiling: hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32 ...
./rebar compile
==> gmt_util (compile)
Compiled src/gmt_util_sup.erl
Compiled src/gmt_elog_policy.erl
Compiled src/gmt_parallel_limit.erl
Compiled src/gmt_pmap.erl
Compiled src/gmt_genutil.erl
Compiled src/gmt_util_app.erl
Compiled src/gmt_tlog_svr.erl
Compiled src/gmt_loop.erl
Compiled src/gmt_sysmon_server.erl
Compiled src/gmt_charset.erl
Compiled src/gmt_cinfo_basic.erl
Compiled src/gmt_edoc.erl
Compiled src/gmt_otp.erl
Compiled src/gmt_erlang.erl
Compiled src/gmt_util.erl
Compiled src/gmt_time.erl
==> riak_err (compile)
Compiled src/riak_err_monitor.erl
Compiled src/riak_err_sup.erl
Compiled src/riak_err_app.erl
Compiled src/riak_err_handler.erl
Compiled src/trunc_io.erl
Compiled src/riak_err_stdlib.erl
==> cluster_info (compile)
Compiled src/cluster_info_ex.erl
Compiled src/cluster_info.erl
==> partition_detector (compile)
Compiled src/partition_detector.erl
Compiled src/partition_detector_sup.erl
Compiled src/partition_detector_mnesia.erl
Compiled src/partition_detector_server.erl
==> congestion_watcher (compile)
Compiled src/congestion_util.erl
Compiled src/congestion_watcher_sup.erl
Compiled src/congestion_watcher.erl
Compiled src/congestion_watcher_server.erl
==> meck (compile)
Compiled src/meck_mod.erl
Compiled src/meck.erl
==> edown (compile)
Compiled src/edown_lib.erl
Compiled src/edown_xmerl.erl
Compiled src/edown_make.erl
Compiled src/edown_doclet.erl
Compiled src/edown_layout.erl
==> asciiedoc (compile)
Compiled src/asciiedown_doclet.erl
Compiled src/asciiedoc_doclet.erl
Compiled src/asciiedoc_lib.erl
==> ubf (compile)
Compiled src/contract_lex.xrl
Compiled src/contract_yecc.yrl
Compiled src/contract_lex.erl
Compiled src/contract_yecc.erl
Compiled src/contract_parser.erl
Compiled src/contract_proto.erl
Compiled src/contract_driver.erl
Compiled src/ubf_plugin_stateless.erl
Compiled src/ubf_plugin_stateful.erl
Compiled src/ubf_driver.erl
Compiled src/contract_manager_tlog.erl
Compiled src/contracts_abnf.erl
Compiled src/contract_manager.erl
Compiled src/proc_socket_server.erl
Compiled src/ebf.erl
Compiled src/ubf_plugin_handler.erl
Compiled src/ubf_plugin_meta_stateless.erl
Compiled src/ubf.erl
Compiled src/proc_utils.erl
Compiled src/ubf_client.erl
Compiled src/ubf_plugin_meta_stateful.erl
Compiled src/ubf_server.erl
Compiled src/ebf_driver.erl
Compiled src/ubf_utils.erl
Compiled src/contracts.erl
==> ubf_thrift (compile)
Compiled src/thrift_contract_parser.erl
Compiled src/ftbf.erl
Compiled src/tbf_driver.erl
Compiled src/ubf_thrift_plugin.erl
Compiled src/ftbf_driver.erl
Compiled src/tbf.erl
==> gdss_brick (compile)
Compiled src/brick_cinfo.erl
Compiled src/brick_sup.erl
Compiled src/brick_ticket.erl
Compiled src/gmt_hlog_local.erl
Compiled src/brick_brick_sup.erl
Compiled src/brick_mboxmon.erl
Compiled src/brick_hash.erl
Compiled src/brick.erl
Compiled src/brick_shepherd.erl
Compiled src/gmt_hlog_common.erl
Compiled src/brick_itimer.erl
Compiled src/brick_pingee.erl
Compiled src/brick_data_sup.erl
Compiled src/gmt_hlog.erl
Compiled src/brick_ets.erl
Compiled src/brick_server.erl
==> gdss_client (compile)
Compiled src/brick_client_sup.erl
Compiled src/brick_client.erl
Compiled src/brick_client_data_sup.erl
Compiled src/brick_simple_stub.erl
Compiled src/brick_simple.erl
Compiled src/brick_simple_client.erl
==> gdss_admin (compile)
Compiled src/brick_admin_cinfo.erl
Compiled src/brick_admin_sup.erl
Compiled src/brick_mon_sup.erl
Compiled src/brick_admin_event_h.erl
Compiled src/brick_migmon.erl
Compiled src/brick_bp.erl
Compiled src/brick_squorum.erl
Compiled src/brick_clientmon.erl
Compiled src/web_admin_sup.erl
Compiled src/brick_chainmon.erl
Compiled src/brick_sb.erl
Compiled src/mod_admin.erl
Compiled src/brick_admin.erl
==> gdss_ubf_proto (compile)
Compiled src/ubf_gdss_plugin.erl
Compiled src/tbf_gdss_plugin.erl
Compiled src/gdss_ubf_proto.erl
Compiled src/gdss_ubf_proto_sup.erl
Compiled src/ubf_gdss_stub_plugin.erl
==> rel (compile)
==> hibari (compile)
generating: hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32 ...
./rebar generate
==> rel (generate)
packaging: hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32 ...
c20fcdd3dff8ff0edf35105b4d2b4608  hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32.tgz
-rw-rw-r-- 1 ubuntu ubuntu       84 Apr 30 03:26 hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32-md5sum.txt
-rw-rw-r-- 1 ubuntu ubuntu 16336110 Apr 30 03:26 hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32.tgz

real    1m26.469s
user    1m41.910s
sys 0m13.550s

make check-package had no error.

ubuntu@esp040:~/hibari/hibari$ make check-package
cleaning: hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32 ...
./rebar clean
...

compiling: hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32 ...
./rebar compile
...

==> hibari (compile)
generating: hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32 ...
./rebar generate
==> rel (generate)
packaging: hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32 ...
cdbc61ffeffecbd03e4413c12bece40a  hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32.tgz
-rw-rw-r-- 1 ubuntu ubuntu       84 Apr 30 03:28 hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32-md5sum.txt
-rw-rw-r-- 1 ubuntu ubuntu 16336039 Apr 30 03:28 hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32.tgz
bootstrapping package: hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32 ...
/bin/sh: 1: ./tmp/hibari/bin/hibari: not found
tar -C ./tmp -xzf ../hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32.tgz
./tmp/hibari/bin/hibari start
./tmp/hibari/bin/hibari-admin bootstrap
ok
checking package: hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32 ...
./tmp/hibari/bin/hibari-admin client-add hibari@127.0.0.1
ok
./tmp/hibari/bin/hibari-admin client-list
['hibari@127.0.0.1']
./tmp/hibari/bin/hibari-admin client-delete hibari@127.0.0.1
ok
./tmp/hibari/bin/hibari checkpoint
ok
./tmp/hibari/bin/hibari stop
ok
@tatsuya6502
Owner

clus.sh in clus tool has a couple of problems with creating hibari user.

  1. It doesn't use $INSTALLER_USER when ssh to the target hosts. I modified clus.sh to use it.
  2. It seems passwd command on Ubuntu doesn't support --stdin option.
/home/tatsuya/workhub/dev/clus% for i in esp040 esp041 esp042 esp043 esp044 esp045 esp046 esp047 ; do ./priv/clus.sh -f init hibari $i ubuntu ; done
user 'hibari@esp040' passwd failed

run ./priv/clus.sh without any arguments for usage
user 'hibari@esp041' passwd failed
...

/home/tatsuya/workhub/dev/clus% ssh ubuntu@esp040
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-40-highbank armv7l)

 * Documentation:  https://help.ubuntu.com/
Last login: Tue Apr 30 04:01:35 2013 from 172.16.181.31
ubuntu@esp040:~$ echo "abc" | sudo passwd --stdin hibari
passwd: unrecognized option '--stdin'
Usage: passwd [options] [LOGIN]

Options:
  -a, --all                     report password status on all accounts
  -d, --delete                  delete the password for the named account
  -e, --expire                  force expire the password for the named account
  -h, --help                    display this help message and exit
  -k, --keep-tokens             change password only if expired
  -i, --inactive INACTIVE       set password inactive after expiration
                                to INACTIVE
  -l, --lock                    lock the password of the named account
  -n, --mindays MIN_DAYS        set minimum number of days before password
                                change to MIN_DAYS
  -q, --quiet                   quiet mode
  -r, --repository REPOSITORY   change password in REPOSITORY repository
  -S, --status                  report password status on the named account
  -u, --unlock                  unlock the password of the named account
  -w, --warndays WARN_DAYS      set expiration warning days to WARN_DAYS
  -x, --maxdays MAX_DAYS        set maximum number of days before password
                                change to MAX_DAYS
@tatsuya6502
Owner

It seems passwd command on Ubuntu doesn't support --stdin option.

I ran this part manually for now.

/home/tatsuya/workhub/dev/clus% ssh ubuntu@esp047
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-40-highbank armv7l)

 * Documentation:  https://help.ubuntu.com/
Last login: Tue Apr 30 04:01:54 2013 from 172.16.181.31
ubuntu@esp047:~$ sudo passwd hibari
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
ubuntu@esp047:~$ logout
Connection to esp047 closed.

/home/tatsuya/workhub/dev/clus% for i in esp040 esp041 esp042 esp043 esp044 esp045 esp046 esp047 ; do ssh-copy-id hibari@$i ; done                  
hibari@esp040's password: 
Now try logging into the machine, with "ssh 'hibari@esp040'", and check in:

  ~/.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

hibari@esp041's password: 
...
@tatsuya6502
Owner

Another problem with clus.sh on Ubuntu is that it doesn't set hibari user's default shell to bash but sh.

ubuntu@esp040:~$ sudo grep hibari /etc/passwd
hibari:x:999:999:hibari node:/usr/local/var/lib/hibari:/bin/sh

So clus-hibari.sh command will fail.

/home/tatsuya/workhub/dev/clus% ./priv/clus-hibari.sh -f init hibari priv/hibari/calxeda.config priv/hibari/hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32.tgz 
sh: 1: source: not found
sh: 1: cd: can't cd to hibari
sh: 1: ./bin/hibari: not found
sh: 1: source: not found
sh: 1: sh: 1: cd: can't cd to hibari
sh: 1: source: not found
sh: 1: cd: sh: 1: source: not found
sh: 1: source: not found
sh: 1: cd: can't cd to hibari
sh: 1: cd: can't cd to hibari
sh: 1: source: not found
sh: 1: cd: can't cd to hibari
sh: 1: source: not found
sh: 1: cd: can't cd to hibari
can't cd to hibari
source: not found
sh: 1: cd: can't cd to hibari
hibari@esp044
hibari@esp045
node esp046 sys.config setup failed

run ./priv/clus-hibari.sh without any arguments for usage
hibari@esp047
hibari@esp043
hibari@esp042
hibari@esp041
hibari@esp040

I manually changed the default shell for hibari to /bin/bash

/home/tatsuya/workhub/dev/clus% ssh ubuntu@esp040
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-40-highbank armv7l)

 * Documentation:  https://help.ubuntu.com/
Last login: Tue Apr 30 04:36:08 2013 from 172.16.181.31
ubuntu@esp040:~$ sudo chsh -s /bin/bash hibari
ubuntu@esp040:~$ logout
Connection to esp040 closed.
...
@tatsuya6502
Owner

OK. Now 8 node hibari cluster on Calxeda Greenbox is up and running.

/home/tatsuya/workhub/dev/clus% ./priv/clus-hibari.sh -f init hibari priv/hibari/calxeda.config priv/hibari/hibari-0.1.10-dev-armv7l-unknown-linux-gnu-32.tgz
hibari@esp041
hibari@esp043
hibari@esp044
hibari@esp045
hibari@esp046
hibari@esp042
hibari@esp047
hibari@esp040
/home/tatsuya/workhub/dev/clus% ./priv/clus-hibari.sh -f start hibari priv/hibari/calxeda.config                                                             
hibari@esp043
hibari@esp042
hibari@esp044
hibari@esp041
hibari@esp047
hibari@esp046
hibari@esp045
hibari@esp040
/home/tatsuya/workhub/dev/clus% ./priv/clus-hibari.sh -f ping hibari priv/hibari/calxeda.config             
hibari@esp040 ... pong
hibari@esp041 ... pong
hibari@esp042 ... pong
hibari@esp043 ... pong
hibari@esp044 ... pong
hibari@esp045 ... pong
hibari@esp046 ... pong
hibari@esp047 ... pong
/home/tatsuya/workhub/dev/clus% ./priv/clus-hibari.sh -f bootstrap hibari priv/hibari/calxeda.config 
ok
hibari@esp040 => hibari@esp040 hibari@esp041 hibari@esp042 hibari@esp043 hibari@esp044 hibari@esp045 hibari@esp046 hibari@esp047
@tatsuya6502
Owner

Forgot to mention. I used kerl to build Erlang/OTP on Calxeda Greenbox.

ubuntu@esp040:~$ kerl status
Available builds:
R15B03-1,r15b03-1
----------
Available installations:
r15b03-1 /home/ubuntu/erlang/r15b03-1
----------
No Erlang/OTP kerl installation is currently active

ubuntu@esp040:~$ cat .kerlrc
KERL_CONFIGURE_OPTIONS="--enable-hipe --enable-smp-support --enable-threads --enable-kernel-poll" 

I put --enable-hipe by mistake, but it seems it got disabled anyway.

hibari@esp040:~$ ./hibari/erts-5.9.3.1/bin/erl
Erlang R15B03 (erts-5.9.3.1) [source] [smp:4:4] [async-threads:0] [kernel-poll:false]

Eshell V5.9.3.1  (abort with ^G)
1> 
User switch command
 --> q
@tatsuya6502
Owner

I just completed a 10-minuts Basho Bench run against 8 nodes Hibari cluster.

{concurrent, 200}.
{duration, 10}.
{mode, max}.
{key_generator, {int_to_bin_bigendian, {uniform_int, 1000000}}}.
{value_generator, {fixed_bin,    1024}}. %%    1 KB
{operations, [{get, 70}
              , {get_witness, 10}
              , {add, 5}
              , {replace, 5}
              , {set, 10}
              %% , {rename, 5}  %% Since Hibari v0.3.0
              , {delete, 5}
             ]}.

The table has chain length = 3 and durable writes (group commit to WAL) = true, so the workload was I/O-bound. Although our Greenbox evaluation machine has only one HDD per node (EnergyCore SoC), it demonstrated a good performance. (EnergyCore can hook up max five HDDs)

$ tail -f current/summary.csv 
...

500.001681, 10.00103, 64948, 64948, 0
510.001195, 9.999514, 65220, 65220, 0
520.000732, 9.999537, 64451, 64451, 0
530.0008, 10.000068, 64949, 64949, 0
540.00066, 9.99986, 64558, 64558, 0
550.000761, 10.000101, 64436, 64436, 0
560.000808, 10.000047, 63315, 63315, 0
570.000745, 9.999937, 61998, 61998, 0
580.003685, 10.00294, 62732, 62732, 0
590.000708, 9.997023, 63885, 63885, 0
600.000733, 10.000025, 58719, 58719, 0
hibari@esp040:~$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  1      0 1148676 231624 2506148    0    0     0     5    8    1  8  6 86  0
 1  1      0 1148244 231624 2506424    0    0     0   808 6412 5426 11  6 64 20
 0  1      0 1148532 231624 2506672    0    0     0   708 6217 4667  8  5 68 19
 0  1      0 1148160 231624 2507000    0    0     0   868 6887 5730 14  4 64 19
 0  1      0 1147844 231624 2507284    0    0     0   928 8495 6522 14  5 62 19
 2  1      0 1147568 231624 2507536    0    0     0   872 7166 5249 11  4 67 18
 2  0      0 1147444 231624 2507828    0    0     0   852 6899 6059 12  5 65 18
 0  1      0 1146916 231624 2508176    0    0     0   920 8759 7058 15  6 62 18
 1  1      0 1146764 231624 2508480    0    0     0   892 8280 6305 11  7 63 19
 0  1      0 1146356 231624 2508760    0    0     0   868 6912 5669 11  5 64 20
 0  1      0 1146052 231624 2509096    0    0     0  1800 7496 5241 11  6 65 19
 0  1      0 1145800 231624 2509340    0    0     0   816 7081 5493 11  6 65 18
 1  1      0 1145428 231624 2509620    0    0     0   812 7116 6589  7  5 70 18
 0  1      0 1145204 231624 2509936    0    0     0   904 7227 5318 11  6 66 18
 2  0      0 1144892 231624 2510268    0    0     0   932 7294 5269 11  6 64 19
 1  1      0 1144420 231624 2510572    0    0     0   904 8900 7147 13  4 64 19
 0  1      0 1144236 231624 2510880    0    0     0   900 6097 4438 11  3 67 19
 0  1      0 1143924 231624 2511164    0    0     0   828 7105 4973 11  4 66 19
 1  0      0 1143728 231624 2511472    0    0     0   804 6936 5326  9  6 67 18
 1  1      0 1143512 231624 2511768    0    0     0   836 6912 5084  9  6 67 18
 0  1      0 1143036 231624 2512044    0    0     0   808 7998 6192  9  6 65 19
 0  1      0 1142444 231624 2512372    0    0     0   796 7387 6326 10  4 68 18
 1  1      0 1142348 231624 2512684    0    0     0   904 7912 5996 13  5 65 17
 1  1      0 1142036 231624 2513012    0    0     0   928 7839 6338 13  6 64 18
 0  1      0 1141552 231624 2513308    0    0     0   900 7688 6450 11  5 65 19
 0  1      0 1141296 231624 2513628    0    0     0   868 8231 6743 14  6 63 17
 1  1      0 1141048 231624 2513908    0    0     0   816 7465 5461  9  5 66 19
 0  1      0 1140484 231624 2514260    0    0     0   780 7393 5753 12  5 65 18
 2  1      0 1140584 231624 2514512    0    0     0   764 7213 5961 16  5 64 15
@tatsuya6502 tatsuya6502 added Task and removed No milestone labels
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.