ciao: Single VM Developer Setup

[mcastelino]

Issues #124 documents how to setup a ciao developer setup on a bare metal system. However this setup is not optimal in some cases.

1. When the user does not want to run the ciao setup on bare metal (i.e wants to run ciao in a VM)
2. When the user runs ciao on bare metal but the hardware cannot support running a VM within a VM efficiently (The CNCI's running in the NN VM)

In this case the ciao should be configured to a special all in one mode where a node has dual role (i.e launcher can be a NN and a CN)

High Level Setup

Host Node (VM)

A clear VM running running on a NATed Virtual network with its own DHCP server.

This NATed Virtual Network network configuration is available by default using virt-manager or virsh on most linux distributions.

1. Controller: Running directly on the host
2. Scheduler: Running directly on the host
3. Compute+Network Node (i.e. CN + NN Launcher): Running directly on the host
4. Workloads: Running directly on the host
5. WebUI: Running directly on the host
6. Openstack Services: Running on the host.

So the host is also the CN, NN, Controller , WebUI and Scheduler (and other openstack services) all run native on the host.

Setup

When the system is functioning the overall setup manifest as follows

As you can see below the CNCI VM's will end up on the NATed network. If this setup is run on a bare metal system the CNCI VM's will end up on the physical network. The Tenant VM's are invisible to host network or the NATed network The Bridge on which the NATed network is created should support hairpin mode and be configured as such.

                _____________________________________________________________________________________
                |                                                                                     |
                |                                                                                     |
                |                                                                                     |
                |                                   [Tenant VMs]                         [CNCI VMs]   |
                |                                                                           ||        |
                |                                                                           ||        |
                |                                                                           ||        |
                |                                                                           ||        |
                |    [scheduler]  [controller] [keystone] [CN+NN Launcher]                  ||        |
                  __________________________________________________________________________||________|
                                                              ||                            ||
                                                              ||                            ||
           ------------------------------------------------------------------------------------------------
                                                  Host NATed Virtual Network (Or physical network)

Changes Needed to ciao components

In addition to the changes needed in #124

1. The ciao launcher should report to the scheduler that is both a NN and a CN node
2. SSNTP should allow an agent to be both a CN and NN
3. The ciao-scheduler should recognize that a node can be both a CN and NN comprehend the same when scheduling CNCI's as well as workloads
4. The Networking library should support creating both CN and NN VNIC interfaces from the same agent.

[markdryan]

@mcastelino Dual roles is already supported by SSNTP as far as I know. The roles are bit masks that can be or'ed together.

The changes to launcher should be minimal and I think could be done before the scheduler changes are made. Nothing should break as long as you don't enable launcher in dual mode. I guess the simplest thing to do would be to add a new value for -network, e.g., dual. I can do this tomorrow.

[mcastelino]

@sameo What is the impact to SSNTP roles. Currently we have two explicit roles when we create certificates. netagent and agent. So it may make sense to eliminate the netagent role and use agent for both roles. The ciao-launcher will send in the role via the bit mask anyway?

-role value
SSNTP role [agent, scheduler, controller, netagent, server, cnciagent] (default Unknown)

[tpepper]

I'll add a developer option to scheduler to enable a non-default relaxed runtime mode where cn and nn would not be exclusive resource types.

[mcastelino]

Added branch where this development will take place https://github.com/01org/ciao/tree/mcastelino/topic/singlemachine

[tpepper]

Initial commits from @sameo enable ssntp.Role as a bitmask. I've got a stack of additional ssntp.Role related patches and the scheduler change for combo AGENT|NETAGENT nodes sitting on master at https://github.com/tpepper/ciao. I'm not sure what the process is...should I just push my couple changes to the mcastelino/topic/singlemachine branch or will @mcastelino cherry pick them in?

[tpepper]

As per @mcastelino I've pushed a rebase plus my changes to the https://github.com/01org/ciao/tree/mcastelino/topic/singlemachine branch.

@sameo can you review especially:
928d521
and
2a71778

[sameo]

@tpepper 2a71778 looks really good, thanks for that.

I have some comments about 928d521:
It makes the server API asymmetric for no specific reasons: Other notifiers may need the client role for the same reasons you mention in the commit message.
So either we:

1. Add the client role to all server notifiers
   or
2. Implement a ssntp.server.ClientRole(uuid string) (Role, error) routine

Although 2 adds one ssntp call to the scheduler implementation, I think it's cleaner than 1. It's generic and can be used in many other places while 1 clutters the API a bit.

[tpepper]

ssntp.server.ClientRole() looks good to me. I'll rework in that direction.

[tpepper]

I think I did this wrong. I think what I want is in the session.srcRole not the server.

[sameo]

@tpepper The peer role is session.destRole. See how we call ntf.DisconnectNotify() from server.go for example.

[tpepper]

Hmm. So srcRole is the server and destRole is the client? If that's always the case, can sssntp.session change s/srcRole/serverRole and s/destRole/clientRole?

Either way some comments might be good there in the code.

[sameo]

@tpepper Both client.go and server.go use sessions and srcRole is the client role in the former and the server role in the latter.
For a session src is the session creator and dest is the destination. I agree it should be better documented, I'll fix that.

[mcastelino]

Single VM with an mock keystone implementation is now fully functional.

• Create a VM with hostname ciao-allinone running on a NATed setup (virtual network 192.168.12.0/24)
• From the host enable hairpin mode for the port on which your VM attaches to the linux bridge

        sudo brctl show
        sudo brctl hairpin <bridge> <port> on

Note: If you shut down your machine or VM (redo this), as virt-manager will not persist this.

• In the VM go get ciao and switch to the single machine branch
• Generate single machine certs

ciao-cert -server -role scheduler -email=ciao-devel@lists.clearlinux.org -organization=Intel -host=ciao-allinone -verify 

ciao-cert -role cnciagent -server-cert ./cert-Scheduler-ciao-allinone.pem -email=ciao-devel@lists.clearlinux.org -organization=Intel -host=ciao-allinone -verify 

ciao-cert -role controller -server-cert ./cert-Scheduler-ciao-allinone.pem -email=ciao-devel@lists.clearlinux.org -organization=Intel -host=ciao-allinone -verify 

ciao-cert -role agent,netagent -server-cert ./cert-Scheduler-ciao-allinone.pem -email=ciao-devel@lists.clearlinux.org -organization=Intel -host=ciao-allinone -verify

• Generate a CNCI VM based on these certs as usual
• Now kick off the scheduler, launcher and controller (here the compute net is the VM's virtual network subnet, 192.168.12.0/24).

sudo -E ./ciao-scheduler --cacert=/home/ciao/certs/CAcert-ciao-allinone.pem --cert=/home/ciao/certs/cert-Scheduler-ciao-allinone.pem --heartbeat --alsologtostderr -v 3

sudo ./ciao-launcher --cacert=/home/ciao/certs/CAcert-ciao-allinone.pem --cert=/home/ciao/certs/cert-CNAgent-NetworkingAgent-ciao-allinone.pem --network=dual --compute-net 192.168.12.0/24 --mgmt-net 192.168.12.0/24 --alsologtostderr -v 3 --disk-limit=false

sudo ./ciao-controller --cacert=/home/ciao/certs/CAcert-ciao-allinone.pem --cert=/home/ciao/certs/cert-Controller-ciao-allinone.pem --single --username=csr --password=hello --httpskey=./controller_key.pem --httpscert=./controller_cert.pem -v 3 -alsologtostderr

• Check the ciao-controller output/logs to determine the URI for the mock keystone

cat /var/lib/ciao/logs/controller/ciao-controller.ERROR
...
E0526 11:15:58.118249    5543 main.go:123] ========================
E0526 11:15:58.118750    5543 main.go:124] Identity URL: http://127.0.0.1:44822
E0526 11:15:58.118774    5543 main.go:125] Please
E0526 11:15:58.118785    5543 main.go:126] export CIAO_IDENTITY=http://127.0.0.1:44822
E0526 11:15:58.118801    5543 main.go:127] ========================

• Setup your environment variables for the ciao-cli with the single machine specific values and the identity information from the log file

export CIAO_CONTROLLER=ciao-allinone
export CIAO_USERNAME=admin
export CIAO_PASSWORD=giveciaoatry
export CIAO_IDENTITY=http://127.0.0.1:44822

[mcastelino]

The functionality is fully implemented in 092cc34

The code needs to be merged into master.

As next steps

• Merge current code into master and delete this branch
• Automate certificate generation for single machine setup
• Automate launch of scheduler, controller and launcher
• Extract the Keystone URI from controller logs
• Automate setting of the ciao env variables

[mcastelino]

All patches except scheduler have been merged into master with PR #196. Once the scheduler changes are pulled in and tested this issue can be closed.

[mcastelino]

Moving documentation to wiki https://github.com/01org/ciao/wiki/HOWTO:-Single-VM-Development-Environment

[tpepper]

On Fri 03 Jun at 17:10:04 -0700 notifications@github.com said:

All patches except scheduler have been merged into master with PR #196. Once
the scheduler changes are pulled in and tested this issue can be closed.

The scheduler functional changes are already in. The only thing not in is
a cleanup for stricter typing of the ssntp.Role, but that's not required.

Tim Pepper -- Linux OS Engineering
Intel Open Source Technology Center

[mcastelino]

@tpepper Some changes from the branch (e.g. multi role handling) are not present in master. For example
https://github.com/01org/ciao/blob/master/ciao-scheduler/scheduler.go#L324
assumes single role (using the case statement)
while the branch implementation checks for all possible roles https://github.com/01org/ciao/blob/mcastelino/topic/singlemachine/ciao-scheduler/scheduler.go#L323

This multi role handling has to be added to master.

[tpepper]

On Mon 06 Jun at 08:22:59 -0700 notifications@github.com said:

@tpepper Some changes from the branch (e.g. multi role handling) are not
present in master. For example
https://github.com/01org/ciao/blob/master/ciao-scheduler/scheduler.go#L324
assumes single role (using the case statement)
while the branch implementation checks for all possible roles https://
github.com/01org/ciao/blob/mcastelino/topic/singlemachine/ciao-scheduler/
scheduler.go#L323

This multi role handling has to be added to master.

In master it's up in the one-higher-level function:

https://github.com/01org/ciao/blob/master/ciao-scheduler/scheduler.go#L340

There the disconnect helper is called for each type that matches.
That function changed from a switch (where only one case would evaluate),
to a series of distinct if-conditionals, so that each matching case
is evaluated.

Tim Pepper -- Linux OS Engineering
Intel Open Source Technology Center

[mcastelino]

Fully implemented and integrated with 89c764f