Make OW run with docker for mac

[style95]

Draft version of documentation to setup OW on a Mac host is added.

I have no idea on documentation convention, PG(?), rebase rule.
I want someone to guide me about these community rules.

I know this branch is conflicting with master branch.
Once I get the review, I will amend them and rebase it.

I tried to minimize the changes and maximize to utilize existing contents.

[rabbah]

Thanks Dominic for this pull request!

The git flow we follow is described here https://github.com/openwhisk/openwhisk/wiki/Contributing:-Git-guidelines; as you said we can deal with that later. For reference PG refers to our Jenkins CI. For documentation it is not necessary usually. Lastly on protocol, have you submitted an Apache CLA?

This would close #1431?

[style95]

Thank you for sharing, Rodric.
I will look into it

Yes I have submitted an Apache CLA, got the response and this would close #1431.

[rabbah]

here and elsewhere: Openwhisk -> OpenWhisk

[rabbah]

dose -> does

[rabbah]

continues -> continuous

[rabbah]

I'll give this a try - we don't have a CI for docker on mac but perhaps we should use this opportunity to set it up; @csantanapr what do you think?

[csantanapr]

Thanks @style95 for the PR
Thanks @rabbah for jumping so quickly on the review.

@style95 where you able to run all test cases ? I recall there some tests that don't work if a special network route it's not created, we run a script currently when using docker-machine

I have to give this a try and see if it works for me on my system.

I would say for now supporting both methods are OK but wonder if in the future we drop docker-machine and stick with docker for Mac.

@rabbah I agree that would be good to have CI for mac, without the need for running a Docker host VM, now the CI job can run on a mac VM

Need to look into getting a couple of VMs from Apache INFRA to start building our Jenkins jobs, wonder if they also have macOS envs.

[style95]

@csantanapr Actually, I just ran gradlew distDocker on my local macbook.

I will rebase my code with the latest master branch and apply the feedback.
After that, I will try to run tests and see what happens.

As of now, if I run gradlew test, I get the following error.

:tests:createKeystore FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/Naver/git/openwhisk/tests/build.gradle' line: 78

* What went wrong:
Execution failed for task ':tests:createKeystore'.
> /Users/Naver/git/openwhisk/whisk.properties (No such file or directory)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 56.276 secs

[csantanapr]

Hi @style95 you need to deploy the system before you can run tests.
Then tests run against a running system.

Follow the instruction how to deploy with ansible first, you can verify by invoking an echo action.
The whisk.properties gets created as part of the deployment.

Then run the tests against the running system.

[style95]

@csantanapr, I rebased the code to the latest master and ran the tests.

From ActionProxyContainerTests, all the tests are being failed.

It looks each test is creating a container and requesting to the container with the container IP address(172.17.0.0/32).
But as you know that ip address is not accessible from a host(Docker for Mac).

We may have few options.

1. Update all the tests to use main docker endpoint ip instead of container IP.
   In this case port should be different with existing components.

2. Make a Mac host be accessible to container IP.
   As of now, I have not concrete ideas, but we may create overlay network with network configuration tool such as flannel.

However, I am not sure these ideas are feasible.

.
.
.
actionContainers.ActionProxyContainerTests > openwhisk/dockerskeleton should run sample without init FAILED
    java.util.concurrent.TimeoutException: Request to //172.17.0.12:8080 could not be completed in time.

actionContainers.ActionProxyContainerTests > openwhisk/dockerskeleton should run sample with init that does nothing FAILED
    java.util.concurrent.TimeoutException: Request to //172.17.0.12:8080 could not be completed in time.

actionContainers.ActionProxyContainerTests > openwhisk/dockerskeleton should respond with 404 for bad run argument FAILED
    java.util.concurrent.TimeoutException: Request to //172.17.0.12:8080 could not be completed in time.

actionContainers.ActionProxyContainerTests > openwhisk/dockerskeleton should fail to run a bad script FAILED
    java.util.concurrent.TimeoutException: Request to //172.17.0.12:8080 could not be completed in time.

actionContainers.ActionProxyContainerTests > openwhisk/dockerskeleton should extract and run a compatible zip exec STANDARD_OUT
    JUnit version is: 4.11

actionContainers.ActionProxyContainerTests > openwhisk/dockerskeleton should extract and run a compatible zip exec FAILED
    java.util.concurrent.TimeoutException: Request to //172.17.0.12:8080 could not be completed in time.

actionContainers.ActionProxyContainerTests > openwhisk/dockerskeleton should run and report an error for script not returning a json object FAILED
    java.util.concurrent.TimeoutException: Request to //172.17.0.12:8080 could not be completed in time.

.
.
.

[csantanapr]

@style95 thanks for checking on tests and reporting results and options.

I suspected that you were going to hit that problem, this is why on using docker-machine we have a script we run to enable the network connection.
https://github.com/openwhisk/openwhisk/blob/master/tools/macos/tweak-dockerhost.sh

MACHINE_NAME=whisk
MACHINE_VM_IP=$(docker-machine ip $MACHINE_NAME)

sudo route -n delete 172.17.0.0/16
sudo route -n -q add 172.17.0.0/16 $MACHINE_VM_IP
netstat -nr |grep 172\.17

Running this script would not fix the problem?

Option 1, is not good as the test needs to remain as is since this is really how it works in reality when the system is deployed on ubuntu production.
I wen this path in the past, as I was not a fan of always running this script, but discover that the test actually is design in this way on purpose, using the private network from invoker container to runtime container.

Not familiar with Option 2, but it sounds that making the network work would be the desire solution.

@rabbah @domdom82 thoughts?

[style95]

@csantanapr Thank you for the opinion.

Regarding tweak-dockerhost.sh script, unfortunately it does not work on docker for mac environment.

That script adds route rule for 172.17.0.0/16 network.
All the packets sent to some containers will be forwarded to docker machine.

In the docker machine, which is linux most of the cases, there is docker0 network,
and it will properly forward the packets to a target container.

In docker-machine environment, docker-machine itself also belongs to container network.
It is a gateway(172.17.0.1) for container network.
So it can forward the packets to container, but in docker for mac environment, it is not possible..

I am looking for the workaround.

By the way, I can configure cli and CRUD/invoke of actions with 172.17.0.1 as an endpoint ip.
Is there any case of accessing directly to components with each container IP? rather than api(nginx)?

[markusthoemmes]

@style95 Yes, tests will try to talk to components directly rather than through the API.

[style95]

@markusthoemmes Thank you to let me know.
Let me clarify my query.

In both local/distributed environment, user can CRUD/invoke of actions.
In local environment, endpoint IP is 172.17.0.1 and distributed environment, it is host IP.
So if there is no case user should directly access to containers with container IP such as 172.17.0.X, it might work.

I am just wondering in which cases user/components are communicating with containers with container IPs directly.
As @csantanapr mentioned above, if this is really how it works in reality, it would be better for me to know the cases in order to find the workaround.

As per my understanding, user/components communicate each other according to the IPs in whisk.properties file.
So I just thought that if test cases also can communicate with the IP in whisk.properties file rather than container IP which is retrieved by docker inspect, it might work.

[csantanapr]

Hi @style95 the network path is required because all the unit tests in
https://github.com/openwhisk/openwhisk/blob/master/tests/src/actionContainers/

The user will never call directly to the containers for /init and /run on port 8080, this will be the invoker.
The tests are there to emulate/mock the invoker calling out /init and /run to test the different runtime containers and that they behave the expected way
https://github.com/openwhisk/openwhisk/blob/master/tests/src/actionContainers/ActionContainer.scala#L139

        val mock = new ActionContainer {
            def init(value: JsValue) = syncPost(ip, 8080, "/init", value)
            def run(value: JsValue) = syncPost(ip, 8080, "/run", value)
        }

I really would like to offer users and contributors to be able to run OpenWhisk on their macs with Docker for Mac.

I proposed that we create a new test suite in gradle for testing on docker for mac that exclude this test cases that are not possible to run because of the network constraints.
Something similar on what we have today for Travis https://github.com/openwhisk/openwhisk/blob/master/tests/build.gradle#L30

task testOnDockerForMac(type: Test) {
    exclude '**/actionContainers*'
}

Contributors can use this gradle tasks to run all tests excepts those. ./gradlew :tests:testOnDockerForMac
If they submit a PR Travis will run the tests for them (They always have the option in addition to run them locally with a ubuntu vm if they are working in that area of the code and to work locally)

And for other Contributors using docker-machine they can continue running the whole suite locally with ./gradlew :tests:test

[csantanapr]

@rabbah @markusthoemmes What you think about proposal on defining a gradle task for docker for mac users?

[markusthoemmes]

@csantanapr not a fan at all. Is it maybe possible to achieve this in a different way?

[csantanapr]

@csantanapr not a fan at all. Is it maybe possible to achieve this in a different way?

I ran out of ideas, that was my last resource.

What do you propose?

[style95]

Actually, I could manually send the /run request to dockerSkeleton container from host.

macbook15:openwhisk style95$ curl -X POST http://172.17.0.19:8080/run
{
  "error": "This is a stub action. Replace it with custom logic."
}

The workaround is as follow.
I run squid http proxy container.

docker run --name squid -d --publish 3128:3128 sameersbn/squid:3.3.8-23

And configured http_proxy as that endpoint.

http_proxy=localhost:3128

This proxy container also belongs to docker0 network in xyhve VM and it can reach to all other containers as well.
So if we send any packet via this container, it will be properly forwarded via docker0 network(172.17.0.0/24)

The problem is however, tests are still failing.
It looks that traffic from tests code does not pass through the proxy.
I am looking for the way to configure proxy in tests code(akka.http.scaladsl.model.HttpRequest).

But if anyone have any idea, kindly share it.
Thanks in advance.

This is only to run the test codes.
It will just run one more container, and make traffic go through that container.

For normal operation on Mac, we don`t need any proxy configuration.

If this method is not enough, I will look for the similar way using vpnkit or hyperkit for more seamless use.
(But I doubt this is feasible.)

[csantanapr]

That was what I was thinking to be able to run those tests, the tests will not be able to reach the internal network of the containers. So the solution will be some how use another container as trampoline/proxy to route the /int and /run to the internal ip and port 8080 of the action container being tested.

[csantanapr]

We could also create a custom mock-invoker container that only role is to route the test http calls, maybe this gets the test closer to reality, since this is how it actually works.

[style95]

@markusthoemmes
I noticed that scope of HttpUtils is protected.

protected[core] class HttpUtils(
    hostname: String,
    timeout: FiniteDuration,
    maxResponse: ByteSize) {

It's not accessible from the class other than core package class.

I think it would be better to move tests/src/tests/scala/actionContainers/ActionContainer.scala to core package in test module.
How do you think?

Other option is, to change the scope of HttpUtills.

[rabbah]

OK to change the scope. I protected is because it has semantics that are peculiar to the invoker but there's a comment to that effect in the class.

[style95]

@rabbah I updated the redo command also.
I have tested with docker-machine also.
It worked as expected.

Kindly check on it.

[style95]

I hope this is reviewed before it becomes staled again..

[rabbah]

off the latest head of this pr, wskdev fresh deployed on mac with no intervention on my part.
i am now running tests locally.

note: rerun gradle eclipse to pull in play references.
edit: successfully ran through a number of test suites including container tests.

[rabbah]

Minor nits.

[rabbah]

add period at the end of the sentence.

[rabbah]

s/docker/Docker/ everywhere except in actual commands.

[rabbah]

s/unix/UNIX/

[rabbah]

The OpenWhisk deployment via Ansible uses the docker0 network interface ...

[rabbah]

docker0.

[rabbah]

why is this removed?

[style95]

With this pr, mac environment is for Docker for Mac, and Ansible can deploy OW on Mac with configurations in local environment.
So I copied group_vars/all from local environment.

AFAIK, Ansible can not deploy OW without that option in docker-machine env, but in mac env, it does not.

[rabbah]

could we use the "local" (ie Ubuntu) hosts?

[style95]

Does local mean Ubuntu hosts?
With Docker for Mac, Ansible will deploy OW without any remote ansible connection.
So I thought ansible_connection should be local.

[rabbah]

Right - this file is the same as https://github.com/apache/incubator-openwhisk/blob/master/ansible/environments/local/hosts

[rabbah]

why is this needed?

[style95]

There are few test cases which depends on this utils and it requires proxy configuration.

[rabbah]

OpenWhisk will natively be running on a Mac host with Docker for Mac.

OpenWhisk can run on a Mac host with [Docker for Mac](link to docker for mac).

[rabbah]

OpenWhisk will be running on virtual machine in which Docker daemon is running.

OpenWhisk can on a Mac using a virtual machine in which Docker daemon is running.

[markusthoemmes]

I thought we agreed on not switching to the play client but rather using the Apache HTTP based client for tests as well to not include another dependency?

[style95]

@markusthoemmes
First, sorry for the confusion.
I should talk to you first.

As you proposed, I was trying to use HttpUtils instead of AkkaHttpUtils in tests.
I found AkkaHttpUtils is being used in ActionContainer.scala.
In current version, it returns (status_code, Option[value]), but in HttpUtils it returns Either[ContainerConnectionError, ContainerResponse].
Both of responses does not include status code.

I could update HttpUtils to return status code as well.
But I am afraid it is not accepted, because it will affect the core logic.
And if I modify response type to Either[ContainerConnectionError, ContainerResponse], it may break semantic in test code.

For example,

it should "abort init with empty response" in {
        val (out, err) = withContainer("badaction") { c =>
            val (code, out) = c.init(initPayload("exit"))
            code shouldBe 500
            out shouldBe empty
        }

        out should include("exit")
        // err stream may not be empty if the proxy did not get a chance to
        // drain the action's out/err streams; skip check on err stream
    }

This code, it wants to check whether it returns 500 response in case of badaciton container.
In some cases, response should be 502.
There are more cases.

it should "report error if zip-encoded action does not include required file" in {
        val srcs = Seq(
            Seq("echo.py") -> """
                |def echo(args):
                |  return { "echo": args }
            """.stripMargin)

        val code = ZipBuilder.mkBase64Zip(srcs)

        val (out, err) = withActionContainer() { c =>
            val (initCode, initRes) = c.init(initPayload(code, main = "echo"))
            initCode should be(502)
        }

        checkStreams(out, err, {
            case (o, e) =>
                o shouldBe empty
                e should include("Zip file does not include")
        })
    }

So if I change the response type, and omit status code, it may change semantic of many tests.

I discussed with @rabbah on Slack.
Since expected code changes are not that simple and small, and this thread is too long already, it would be better to open a separate pr iff you approve it.

[style95]

It is rebased from the latest master again, and all the conflicts have been resolved.

[markusthoemmes]

Sounds good to me, thanks for clarification 👍

[rabbah]

We will try to further improve this in the future to use local for docker for mac, and further strip the play dependence. Thanks for being patient and working this out!