Erlang implementation of the libvirtd remote protocol
Erlang Other
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin mk_libvirt_constants: tokenize define Sep 9, 2017
include
priv Rename the test VM to prevent clashes with vert Sep 3, 2012
src Upgrade to libvirt 1.3.1 Sep 6, 2017
test
.gitignore
Makefile makefile: fix typer target Dec 10, 2016
README.md README: remove ancient openwrt instructions Sep 10, 2017
elvis.config
rebar.config Add dialyzer warnings Oct 21, 2016
rebar.lock
start.sh

README.md

Erlang implementation of the libvirtd remote protocol.

For an Erlang binding to the C libvirt interface, see:

https://github.com/msantos/erlang-libvirt

WARNING

remote_protocol.x contains this warning:

(1) The protocol is internal and may change at any time, without
notice.  Do not use it.  Instead link to libvirt and use the remote
driver.

http://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=src/remote/remote_protocol.x;hb=HEAD

However, see the section GENERATING THE REMOTE PROTOCOL MODULE below for instructions on recompiling the XDR protocol spec if any changes occur.

The RPC protocol is documented here:

http://libvirt.org/internals/rpc.html

For the remote support documentation:

http://libvirt.org/remote.html

The version of remote_protocol.x used was taken from libvirt master at 1.3.1 (commit 8fd68675e2b5eed5b2aae636544a0a80f9fc70e9).

HOW TO BUILD IT

make

See GENERATING THE REMOTE PROTOCOL MODULE to rebuild the XDR protocol parser.

TESTING EVERYTHING WORKS

To quickly test everything works, the libvirtd test driver can be used with bin/verx, an escript that provides a simple command line interface to the verx library.

You'll have to set up the ERL_LIBS environment variable first, e.g., if verx is checked out in ~/src:

export ERL_LIBS=$ERL_LIBS:~/src

Then run:

bin/verx

To list the test VMs:

bin/verx list --uri test:///default

To retrieve the test virtual machine configuration:

bin/verx dumpxml test --uri test:///default

To create the example VM (a no-op with the test driver):

bin/verx create priv/example.xml --uri test:///default

To see all the VMs (if you have TLS set up):

bin/verx list --all --transport verx_client_tls --uri test:///default

To screenshot the test VM:

bin/verx screenshot test --uri test:///default

The test driver will return an error if console access is requested. To connect to an actual VM's console using the Unix transport:

bin/verx console localvm # control-C to exit

HOW TO USE IT

libvirt documentation

See http://libvirt.org/html/libvirt-libvirt.html

DATA TYPES

verx_transport()

    Reference to the underlying transport and transport handler.

unix_socket() = string() | binary()

    Path to Unix socket.

verx

verx:Call(Ref) -> ok | {ok, Payload} | {error, Error}
verx:Call(Ref, Arg) -> ok | {ok, Payload} | {error, Error}

    Types   Call = [connect_open, connect_close, connect_list_domain, ...]
            Ref = verx_transport()
            Arg = [remote_protocol_args()]
            Payload = [remote_protocol_ret()]
            Error = [ posix() | libvirt() ]

verx has a large number of functions (283). See verx.erl or the
exports in verx:module_info() for a list.

Understanding the arguments for a remote protocol call takes some
work.  For example, for verx:domain_define_xml/2, here are some
places to look at:

    * check verx.erl for the arity

    * check remote_protocol_xdr.erl for the argument format. The
      parsing function is prefaced with "enc_remote_" and ends with
      "_args":

        enc_remote_domain_define_xml_args/1

    * check the XDR protocol file, remote_protocol.x:

        struct remote_domain_define_xml_args {
            remote_nonnull_string xml;
        };

    * look at the libvirt documentation. Generally the libvirt
      counterpart is camelcased and prefaced with "vir":

        virDomainDefineXML

Similarly, for the call return values, search for the
suffix "_ret", e.g., dec_remote_domain_define_xml_ret and
remote_domain_define_xml_ret.

verx_client

verx_client:start(Opt) -> {ok, Ref} | {error, posix()}

    Types   Opt = [ Options ]
            Options = {transport, Transport}

                  % Unix socket
                | {path, unix_socket()}

                  % TCP and TLS
                | {host, ip_address()}
                | {port, uint16()}

                  % TLS
                | {cacert, path()}
                | {cert, path()}
                | {key, path()}
                | {depth, integer()}
                | {password, string()}
                | {ciphers, ciphers()}

            Transport = verx_client_unix
                | verx_client_tcp
                | verx_client_tls

    RPC transport layer, supports Unix sockets, TCP and TLS (IPv4
    and IPV6).

    Options depend on the underlying transport mechanism.

verx_client:stop(Ref) -> ok

    Closes the transport socket.

verx_client:recv(Ref) -> {ok, Buf} | {error, posix()}

    Types   Ref = verx_transport()
            Buf = [binary()]

    Returns streamed data. The stream must first be prepared
    by making the appropriate remote protocol call, e.g.,
    verx:domain_snapshot/2.

verx_client_unix

verx_client_tcp

verx_client_tls

verx_rpc

EXAMPLES

OPEN A CONNECTION TO LIBVIRTD

% Connect to the libvirtd socket
{ok, Ref} = verx_client:start(),

% libvirt remote procotol open message
% by default to qemu:///system
ok = verx:connect_open(Ref),

% send a close message
ok = verx:connect_close(Ref),

% send a remote protocol open message
%  connecting to lxc containers
ok = verx:connect_open(Ref, ["lxc:///", 0]),

% close and stop the transport
ok = verx:connect_close(Ref),
ok = verx_client:stop(Ref).

% open a TLS connection on the default port
CACert = "/tmp/cert/cacert.pem",
Cert = "/tmp/cert/clientcert.pem",
Key = "/tmp/cert/clientkey.pem",

{ok, Ref} = verx_client:start([
        {transport, verx_client_tls},
        {cacert, CACert},
        {cert, Cert},
        {key, Key}
        ]).

CREATING A DOMAIN

-module(crvm).
-export([file/0]).

file() ->
    file("priv/example.xml").
file(Path) ->
    % Connect to the libvirtd socket
    {ok, Ref} = verx_client:start(),

    % libvirt remote procotol open message
    ok = verx:connect_open(Ref),

    {ok, XML} = file:read_file(Path),

    % Domain is defined but not running
    {ok, [Domain]} = verx:domain_define_xml(Ref, [XML]),

    % Start the VM
    ok = verx:domain_create(Ref, [Domain]),

    {ok, [Active] = verx:connect_num_of_domains(Ref),
    io:format("Active Domains: ~p~n", [Active]),

    % Send a protocol close message
    ok = verx:connect_close(R),

    % Close the socket
    ok = verx_client:stop(R),

    {ok, Domain}.

To list the VMs:

-module(lsvm).
-export([ls/0]).

ls() ->
    {ok, Ref} = verx_client:start(),
    ok = verx:connect_open(Ref),

    {ok, [NumDef]} = verx:connect_num_of_defined_domains(Ref),

    {ok, [NumRun]} = verx:connect_num_of_domains(Ref),

    {ok, [Shutoff]} = verx:connect_list_defined_domains(Ref, [NumDef]),
    {ok, [Running]} = verx:connect_list_domains(Ref, [NumRun]),

    {ok, [{running, info(Ref, Running)},
             {shutoff, info(Ref, Shutoff)}]}.

info(Ref, Domains) ->
    [ begin
        {ok, [{Name, UUID, Id}]} = verx:domain_lookup_by_id(Ref, [N]),
        {Name, [{uuid, UUID}, {id, Id}]}
      end || N <- Domains ].

To shutdown the VM:

% Get the domain resource
lookup(Ref, Id) when is_integer(Id) ->
    {ok, [Domain]} = verx:domain_lookup_by_id(Ref, [Id]),
    {ok, Domain};

lookup(Ref, Name) when is_binary(Name) ->
    {ok, [Domain]} = verx:domain_lookup_by_name(Ref, [Name]),
    {ok, Domain}.

halt(Ref, Domain) ->
    % shutdown only works if acpid is installed in the VM
    ok = verx:domain_shutdown(R, [Domain]),
    verx:domain_destroy(Ref, [Domain]).

To remove the VM, undefine it:

    verx:domain_undefine(Ref, [Domain])

SUSPENDING AND RESUMING A DOMAIN

This example provides the Erlang equivalent of a Python script to manipulate a running domain. The example was taken from:

http://www.ibm.com/developerworks/linux/library/l-libvirt/

-module(ex6).

-export([start/0, states/2]).

start() ->
    {ok, Ref} = verx_client:start(),
    ok = verx:connect_open(Ref),

    {ok, [Num]} = verx:connect_num_of_domains(Ref),
    {ok, [Ids]} = verx:connect_list_domains(Ref, [Num]),

    [ states(Ref, Id) || Id <- Ids ],
    ok.

states(Ref, Id) ->
    {ok, [Domain]} = verx:domain_lookup_by_id(Ref, [Id]),

    % return value of domain_get_info from remote_protocol.x:
    %
    % struct remote_domain_get_info_ret {
    %   unsigned char state;
    %   unsigned hyper maxMem;
    %   unsigned hyper memory;
    %   unsigned short nrVirtCpu;
    %   unsigned hyper cpuTime;
    % };

    io:format("running: ~p~n", [verx:domain_get_info(Ref, [Domain])]),

    ok = verx:domain_suspend(Ref, [Domain]),
    io:format("suspended: ~p~n", [verx:domain_get_info(Ref, [Domain])]),

    ok = verx:domain_resume(Ref, [Domain]),
    io:format("resumed: ~p~n", [verx:domain_get_info(Ref, [Domain])]),

    ok = verx:domain_shutdown(Ref, [Domain]),
    io:format("shutdown: ~p~n", [verx:domain_get_info(Ref, [Domain])]),

    ok = verx:domain_destroy(Ref, [Domain]),
    io:format("destroyed: ~p~n", [verx:domain_get_info(Ref, [Domain])]).

RETRIEVING HYPERVISOR INFORMATION

Here is some code to retrieve information about the hypervisor, similar to the example in the Ruby libvirt documentation (http://libvirt.org/ruby/examples/node_info.rb):

-module(node_info).
-export([start/0]).

start() ->
    {ok, Ref} = verx_client:start(),
    ok = verx:connect_open(Ref),

    [ begin
            Reply = case Proc of
                {Call, Arg} -> verx:Call(Ref, Arg);
                Call -> verx:Call(Ref)
            end,
            result(Proc, Reply)
      end || Proc <- [
                node_get_info,
                {node_get_cells_free_memory, [0, 100]},
                connect_get_version,
                connect_get_lib_version,
                connect_get_hostname,
                connect_get_uri,
                node_get_free_memory,
                node_get_security_model,
                connect_is_secure,
                connect_get_capabilities
                ] ],

    ok = verx:connect_close(Ref),
    verx_client:stop(Ref).

result(Call, {ok, N}) ->
    error_logger:info_report([{call, Call}] ++ N);
result(Call, {error, _Error} = N) ->
    error_logger:error_report([{call, Call}] ++ N).

SYSTEM CONSOLE

The VM system console can be accessed using any of the transports.

% Connect to libvirtd using the Unix socket
1> {ok, Ref} = verx_client:start().
{ok,<0.43.0>}

% Open a remote protocol session to the Linux containers hypervisor
2> verx:connect_open(Ref, ["lxc:///", 0]).
ok

% Get a domain reference
3> {ok, [Domain]} = verx:domain_lookup_by_name(Ref, [<<"lxc-1">>]).
{ok,[{<<"lxc-1">>,
     <<150,162,91,134,54,66,203,130,29,224,244,242,121,45,5,118>>,
       19586}]}

% Open the console. The arguments are:
%   Domain
%   Device name : string() or void (NULL)
%   Flags : integer()
4> verx:domain_open_console(Ref, [Domain, void, 0]).

% Send a message to the console, check the results with
% flush()

% Start up Erlang ...
5> verx_client:send(Ref, [<<"erl\n">>]).

6> verx_client:send(Ref, [<<"spawn(fun() -> io:format(\"Erlang process in an Erlang VM in a Linux VM in an Erlang process!\") end).\n">>]).

% Receive the message back from the console
8> verx_client:recv(Ref).
{ok,<<"Erlang process in an Erlang VM in a Linux VM in an Erlang process!">>}

TAKING A SCREENSHOT

An example of using the libvirt stream interface to capture an image of the VM console:

-module(ss).
-export([host/1]).

host(Name) when is_list(Name) ->
    host(list_to_binary(Name));
host(Name) when is_binary(Name) ->
    {ok, Ref} = verx_client:start(),
    ok = verx:connect_open(Ref),

    {ok, [Domain]} = verx:domain_lookup_by_name(Ref, [Name]),

    Screen = 0,
    Flags = 0,

    {ok, [Mime]} = verx:domain_screenshot(Ref, [Domain, Screen, Flags]),

    Ext = case Mime of
        <<"image/x-portable-pixmap">> -> <<".ppm">>;
        _ -> <<".screen">>
    end,

    {ok, Buf} = verx_client:recvall(Ref),

    File = <<Name/binary, Ext/binary>>,
    ok = file:write_file(File, Buf),

    {ok, Mime, File}.

CREATING LINUX CONTAINERS

This example will generate many Linux containers (LXC) attached to a bridge (br0).

-module(clxc).
-export([start/2, start/3, create/2, template/2]).

start(Prefix, Num) ->
    {ok, Ref} = verx_client:start(),
    ok = verx:connect_open(Ref, ["lxc:///", 0]),
    start(Ref, Prefix, Num).

start(_Ref, _Prefix, 0) ->
    ok;

start(Ref, Prefix, Num) ->
    Name = Prefix ++ integer_to_list(Num),

    <<Bytes:3/bytes, _/binary>> = erlang:md5(Name),
    Macaddr = "52:54:00:" ++ string:join([ httpd_util:integer_to_hexlist(N)
        || <<N:8>> <= Bytes ], ":"),

    XML = template(Name, Macaddr),
    ok = create(Ref, XML),

    start(Ref, Prefix, Num-1).

create(Ref, XML) ->
    {ok, [Domain]} = verx:domain_define_xml(Ref, [XML]),
    verx:domain_create(Ref, [Domain]).

template(Name, Macaddr) ->
"<domain type='lxc'>
    <name>" ++ Name ++ "</name>
    <memory>102400</memory>
    <os>
        <type>exe</type>
        <init>/bin/sh</init>
    </os>
    <devices>
        <console type='pty'/>
        <interface type='bridge'>
            <mac address='" ++ Macaddr ++ "'/>
            <source bridge='br0'/>
        </interface>
    </devices>
</domain>".

GENERATING THE REMOTE PROTOCOL MODULE

To create the remote_protocol_xdr.erl from a remote_protocol.x file:

  1. Copy remote_protocol.x to priv

  2. Run:

     make clean; make
    

If there are any errors, read through bin/mk_remote_protocol.escript.

VERX CLIENT

verx is a simple command line client similar to virsh. To use verx, the ERL_LIBS environment variable must point to the directory containing the verx repository:

export ERL_LIBS=$ERL_LIBS:~/src
export PATH=$PATH:~/src/verx/bin

Running verx without any options will return the list of commands.

All verx commands can take some options:

--uri : URI supported by libvirt (default: qemu:///system)
--transport : (default: verx_client_unix)
    verx_client_unix
    verx_client_tcp
    verx_client_tls

For TCP and TLS transports:

--host : hostname
--port : port

For the TLS transport:

--cacert : path to CA cert (default: /etc/pki/CA/cacert.pem)
--cert : path to client cert (default: /etc/pki/libvirt/clientcert.pem)
--depth : cert validation depth (default: 1)

Examples:

# List all defined Qemu/KVM instances through the libvirtd Unix socket
verx list --all

# List running LXC instances
verx list --uri=lxc:///

# Dump the configuration of a KVM using the TLS transport over IPv6
verx dumpxml myvm --transport verx_client_tls --host ::1

# Access the console of a container over TLS/IPv6
# Use ctl-C to exit
verx console mylxc --uri lxc:/// --transport verx_client_tls --host ::1

TODO

  • verx_client_tls
    • single byte received before packet (works if thrown away)