Basic port-test capabilities before trying to bring up router, fixes #…

…126, fixes #393 (#483)

* Basic port-test capabilities before trying to bring up router

* Add troubleshooting docs

* Remove double-notification about bad router, fixes #393

cmd/ddev/cmd/start.go

@@ -33,7 +33,7 @@ provide a working environment for development.`,
 func appStart() {
 	app, err := platform.GetActiveApp("")
 	if err != nil {
-		util.Failed("Failed to start: %s", err)
+		util.Failed("Failed to start %s, err: %v", app.GetName(), err)
 	}
 
 	fmt.Printf("Starting environment for %s...\n", app.GetName())

docs/developers/surf-testmachine-setup.md

@@ -1,6 +1,6 @@
 <h1>Surf Test Machine Setup</h1>
 
-We are using [surf](https://github.com/surf-build/surf) for Windows and OSX testing. This is a very simple test setup that uses on-premise machines that have to be configured correctly to run.
+We are using [surf](https://github.com/surf-build/surf) for Windows and macOS testing. This is a very simple test setup that uses on-premise machines that have to be configured correctly to run.
 
 Before beginning:
 1. Obtain a GITHUB_TOKEN with repo and gist privileges (and for a user who has write/push privs on the repo). Currently this GITHUB_TOKEN is associated with drud-test-machine-account and the token is shared in lastpass.
@@ -28,7 +28,7 @@ surf-install -e DDEV_PANTHEON_API_TOKEN -n surf-windows-ddev -c "surf-run -r htt
 ```
 where `surf-windows-ddev` is the name of the job on the windows machine (it's free-form) and `surf-windows` is the identifier used on github when the build changes statuses; it's also free-form. **Note that current versions of surf-install will fail claiming that schtasks failed, but it seems to work fine anyway. See [issue](https://github.com/surf-build/surf/issues/64).**
 
-### OSX Test Machine Setup
+### macOS Test Machine Setup
 
 1. Install [homebrew](https://brew.sh/)
 2. Install golang/git/docker with `brew install golang git docker`
@@ -48,4 +48,4 @@ surf-install -e DDEV_PANTHEON_API_TOKEN -n surf-darwin-ddev -c "surf-run -r http
 ```
 where `surf-darwin-ddev` is the name of the job on the mac machine (it's free-form) and `surf-darwin` is the identifier used on github when the build changes statuses; it's also free-form.
 
-9. Reboot the machine and do a test run. Alternatively, test with `surf-build -r https://github.com/drud/ddev -s <some_sha>`
+9. Reboot the machine and do a test run. Alternatively, test with `surf-build -r https://github.com/drud/ddev -s <some_sha>`

docs/users/cli-usage.md

@@ -341,4 +341,4 @@ You can stop a site's containers without losing data by using `ddev stop` in the
 You can remove a site's containers by running `ddev remove` in the working directory of the site. You can also remove any running site's containers by providing the site name as an argument, e.g. `ddev remove <sitename>`. **Note:** `ddev remove` is destructive. It will remove all containers for the site, destroying database contents in the process. Your project code base and files will not be affected.
 
 ## ddev Command Auto-Completion
-ddev bash auto-completion is available. If you have installed ddev via homebrew (on OSX) it will already be installed. Otherwise, you can download the [latest release](https://github.com/drud/ddev/releases) tarball for your platform and the ddev_bash_completions.sh inside it can be installed wherever your bash_completions.d is. For example, `cp ddev_bash_completions.sh /etc/bash_completion.d/ddev`
+ddev bash auto-completion is available. If you have installed ddev via homebrew (on macOS) it will already be installed. Otherwise, you can download the [latest release](https://github.com/drud/ddev/releases) tarball for your platform and the ddev_bash_completions.sh inside it can be installed wherever your bash_completions.d is. For example, `cp ddev_bash_completions.sh /etc/bash_completion.d/ddev`

docs/users/step-debugging.md

@@ -8,7 +8,7 @@ All IDEs basically work the same: They listen on a port and react when they're c
 
 **Key facts:**
 * The debug server port on the IDE must be set to port 11011. Although the xdebug default is port 9000, that port often has conflicts for PHP developers, so 11011 is used with ddev.
-* An IP-address *alias* of 172.28.99.99 must be added to your workstation host's loopback address. On MacOS this is done with the command `sudo ifconfig lo0 alias 172.28.99.99`. **This must currently be done after each reboot.**
+* An IP-address *alias* of 172.28.99.99 must be added to your workstation host's loopback address. On macOS this is done with the command `sudo ifconfig lo0 alias 172.28.99.99`. **This must currently be done after each reboot.**
 
 For more background on XDebug see [XDebug documentation](https://xdebug.org/docs/remote). The intention here is that one won't have to understand XDebug to do debugging.
 

docs/users/troubleshooting.md

@@ -0,0 +1,47 @@
+<h1>Troubleshooting</h1>
+
+Things might go wrong!
+
+<a name="unable-listen"></a>
+## Webserver ports are already occupied by another webserver
+
+If you get a message from ddev about a port conflict on port 80 or 443, like this:
+
+```
+Failed to start yoursite: Unable to listen on required ports, Localhost port 80 is in use
+```
+
+it means that you have another webserver listening on port 80 (or 443, or both), and it needs to be stopped so that ddev can access the port. 
+
+Probably the most common reason for this is that Apache is running locally. It can often be stopped gracefully (but temporarily) with:
+
+```
+sudo apachectl stop
+```
+
+**Common tools that use port 80:**
+
+There are many processes that could be using port 80. Here are some of the common ones and how to stop them:
+
+* MAMP (macOS): [Stop MAMP](http://documentation.mamp.info/en/MAMP-Mac/Preferences/Start-Stop/)
+* Apache: Temporarily stop with `sudo apachectl stop`, permanent stop depends on your environment.
+* nginx (macOS Homebrew): `sudo brew services stop nginx`
+or `sudo launchctl stop homebrew.mxcl.nginx`
+* nginx (Ubuntu): `sudo service nginx stop`
+* apache (often named "httpd") (many environments): `sudo apachectl stop` or on Ubuntu `sudo service apache2 stop`
+* vpnkit (macOS): You likely have a docker container bound to port 80, do you have containers up for Kalabox or another docker-based development environment? If so, stop the other environment.
+* Kalabox: If you have previously used Kalabox try running `kbox poweroff`
+
+To dig deeper, you can use a number of tools to find out what process is listening. On macOS and Linux, try the lsof tool:
+
+```
+$ sudo lsof -i :80 -sTCP:LISTEN
+COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
+nginx   1608 www-data   46u  IPv4  13913      0t0  TCP *:http (LISTEN)
+nginx   5234     root   46u  IPv4  13913      0t0  TCP *:http (LISTEN)
+```
+
+As you see, the command that's running is listed, and its pid. You then need to use the appropriate technique to stop the other server. 
+
+
+We welcome your [suggestions](https://github.com/drud/ddev/issues/new) based on other issues you've run into and your troubleshooting technique.

pkg/plugins/platform/local.go

@@ -970,6 +970,7 @@ func GetActiveAppRoot(siteName string) (string, error) {
 }
 
 // GetActiveApp returns the active App based on the current working directory or running siteName provided.
+// To use the current working directory, siteName should be ""
 func GetActiveApp(siteName string) (App, error) {
 	app, err := GetPluginApp("local")
 	if err != nil {

pkg/plugins/platform/local_test.go

@@ -12,6 +12,7 @@ import (
 
 	"github.com/drud/ddev/pkg/ddevapp"
 	"github.com/drud/ddev/pkg/dockerutil"
+	"github.com/drud/ddev/pkg/exec"
 	"github.com/drud/ddev/pkg/fileutil"
 	"github.com/drud/ddev/pkg/plugins/platform"
 	"github.com/drud/ddev/pkg/testcommon"
@@ -478,6 +479,63 @@ func TestDescribeStopped(t *testing.T) {
 	}
 }
 
+// TestRouterPortsCheck makes sure that we can detect if the ports are available before starting the router.
+func TestRouterPortsCheck(t *testing.T) {
+	assert := asrt.New(t)
+
+	// First, stop any sites that might be running
+	app, err := platform.GetPluginApp("local")
+	assert.NoError(err)
+
+	// Stop all sites, which should get the router out of there.
+	for _, site := range TestSites {
+		switchDir := site.Chdir()
+
+		testcommon.ClearDockerEnv()
+		err := app.Init(site.Dir)
+		assert.NoError(err)
+
+		err = app.Stop()
+		assert.NoError(err)
+
+		switchDir()
+	}
+
+	// Now start one site, it's hard to get router to behave without one site.
+	site := TestSites[0]
+	testcommon.ClearDockerEnv()
+
+	app, err = platform.GetActiveApp(site.Name)
+	if err != nil {
+		t.Fatalf("Failed to GetActiveApp(%s), err:%v", site.Name, err)
+	}
+	err = app.Start()
+	assert.NoError(err, "app.Start(%s) failed, err: %v", app.GetName(), err)
+
+	// Stop the router using code from platform.StopRouter().
+	// platform.StopRouter can't be used here because it checks to see if containers are running
+	// and doesn't do its job as a result.
+	dest := platform.RouterComposeYAMLPath()
+	err = dockerutil.ComposeCmd([]string{dest}, "-p", platform.RouterProjectName, "down", "-v")
+	assert.NoError(err, "Failed to stop router using docker-compose, err=%v", err)
+
+	// Occupy port 80 using docker busybox trick, then see if we can start router.
+	// This is done with docker so that we don't have to use explicit sudo
+	containerId, err := exec.RunCommand("sh", []string{"-c", "docker run -d -p80:80 --rm busybox:latest sleep 100 2>/dev/null"})
+	if err != nil {
+		t.Fatalf("Failed to run docker command to occupy port 80, err=%v output=%v", err, containerId)
+	}
+	containerId = strings.TrimSpace(containerId)
+
+	// Now try to start the router. It should fail because the port is occupied.
+	err = platform.StartDdevRouter()
+	assert.Error(err, "Failure: router started even though port 80 was occupied")
+
+	// Remove our dummy busybox docker container.
+	out, err := exec.RunCommand("docker", []string{"rm", "-f", containerId})
+	assert.NoError(err, "Failed to docker rm the port-occupier container, err=%v output=%v", err, out)
+}
+
 // TestCleanupWithoutCompose ensures app containers can be properly cleaned up without a docker-compose config file present.
 func TestCleanupWithoutCompose(t *testing.T) {
 	assert := asrt.New(t)

pkg/plugins/platform/router.go

@@ -11,13 +11,17 @@ import (
 
 	"strings"
 
+	"net"
+
 	"github.com/drud/ddev/pkg/dockerutil"
 	"github.com/drud/ddev/pkg/util"
 	"github.com/drud/ddev/pkg/version"
 	"github.com/fatih/color"
+	"github.com/fsouza/go-dockerclient"
 )
 
-const routerProjectName = "ddev-router"
+// RouterProjectName is the "machine name" of the router docker-compose
+const RouterProjectName = "ddev-router"
 
 // RouterComposeYAMLPath returns the full filepath to the routers docker-compose yaml file.
 func RouterComposeYAMLPath() string {
@@ -36,7 +40,7 @@ func StopRouter() error {
 
 	if !containersRunning {
 		dest := RouterComposeYAMLPath()
-		return dockerutil.ComposeCmd([]string{dest}, "-p", routerProjectName, "down", "-v")
+		return dockerutil.ComposeCmd([]string{dest}, "-p", RouterProjectName, "down", "-v")
 	}
 	return nil
 }
@@ -76,14 +80,21 @@ func StartDdevRouter() error {
 	_, err = f.WriteString(doc.String())
 	util.CheckErr(err)
 
+	container, err := findDdevRouter()
+	// If we have a router running, we don't have to stop and start it.
+	if err != nil || container.State != "running" {
+		err = CheckRouterPorts()
+		if err != nil {
+			return fmt.Errorf("Unable to listen on required ports, %v,\nTroubleshooting suggestions at https://ddev.readthedocs.io/en/latest/users/troubleshooting/#unable-listen", err)
+		}
+	}
+
 	// run docker-compose up -d in the newly created directory
-	err = dockerutil.ComposeCmd([]string{dest}, "-p", routerProjectName, "up", "-d")
+	err = dockerutil.ComposeCmd([]string{dest}, "-p", RouterProjectName, "up", "-d")
 	if err != nil {
 		return fmt.Errorf("failed to start ddev-router: %v", err)
 	}
 
-	fmt.Println("Starting service health checks...")
-
 	// ensure we have a happy router
 	label := map[string]string{"com.docker.compose.service": "ddev-router"}
 	err = dockerutil.ContainerWait(containerWaitTimeout, label)
@@ -94,18 +105,32 @@ func StartDdevRouter() error {
 	return nil
 }
 
+// findDdevRouter usees FindContainerByLabels to get our router container and
+// return it
+func findDdevRouter() (docker.APIContainers, error) {
+	containerQuery := map[string]string{
+		"com.docker.compose.service": RouterProjectName,
+	}
+	container, err := dockerutil.FindContainerByLabels(containerQuery)
+	if err != nil {
+		return docker.APIContainers{}, fmt.Errorf("Failed to execute findContainersByLabels, %v", err)
+	}
+	return container, nil
+}
+
 // PrintRouterStatus outputs router status and warning if not
 // running or healthy, as applicable.
+// An easy way to make these ports unavailable in order to test this is:
+// sudo netcat -l -p 80  (brew install netcat)
 func PrintRouterStatus() string {
 	var status string
 
 	badRouter := "\nThe router is not currently running. Your sites are likely inaccessible at this time.\nTry running 'ddev start' on a site to recreate the router."
 
-	label := map[string]string{"com.docker.compose.service": "ddev-router"}
-	container, err := dockerutil.FindContainerByLabels(label)
+	container, err := findDdevRouter()
 
 	if err != nil {
-		status = color.RedString(SiteNotFound) + badRouter
+		status = color.RedString(SiteNotFound)
 	} else {
 		status = dockerutil.GetContainerHealth(container)
 	}
@@ -168,3 +193,22 @@ func determineRouterPorts() []string {
 	}
 	return routerPorts
 }
+
+// CheckRouterPorts tries to connect to ports 80/443 as a heuristic to find out
+// if they're available for docker to bind to. Returns an error if either one results
+// in a successful connection.
+func CheckRouterPorts() error {
+	// TODO: When we allow configurable ports, we'll want to use an array of configured ports here.
+	for _, port := range []int{80, 443} {
+		target := fmt.Sprintf("127.0.0.1:%d", port)
+		conn, err := net.Dial("tcp", target)
+		// We want an error (inability to connect), that's the success case.
+		// If we don't get one, return err. This will normally be "getsockopt: connection refused"
+		// For simplicity we're not actually studying the err value.
+		if err == nil {
+			_ = conn.Close()
+			return fmt.Errorf("Localhost port %d is in use", port)
+		}
+	}
+	return nil
+}