Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update from source #2

Merged
merged 63 commits into from
May 6, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
0fcd354
Bump github.com/gin-gonic/gin from 1.5.0 to 1.6.0
dependabot-preview[bot] Mar 23, 2020
6888754
Merge pull request #167 from testcontainers/dependabot/go_modules/git…
gianarb Mar 23, 2020
485c09c
Bump github.com/gin-gonic/gin from 1.6.0 to 1.6.1
dependabot-preview[bot] Mar 24, 2020
153da02
Merge pull request #168 from testcontainers/dependabot/go_modules/git…
gianarb Mar 24, 2020
7cca004
Bump github.com/gin-gonic/gin from 1.6.1 to 1.6.2
dependabot-preview[bot] Mar 27, 2020
90c3f43
Merge pull request #170 from testcontainers/dependabot/go_modules/git…
gianarb Mar 27, 2020
533329b
#173: Reusing existing reaper for subsequent container/network create…
ikolomiyets Apr 5, 2020
a80d5c8
Adding a property to the ContainerRequest that forces Testcontainers …
ikolomiyets Apr 5, 2020
b5ed60c
Merge pull request #175 from ikolomiyets/force-pull
gianarb Apr 6, 2020
8d295a6
The underlying issue is apparently more complex. I think, that partia…
ikolomiyets Apr 6, 2020
dfa345a
chore(vendor): update docker engine and uuid
Apr 10, 2020
36067b9
Merge pull request #177 from testcontainers/bump/vendor-cleanup
gianarb Apr 10, 2020
3a061a0
fix(uuid): wrong replacement for uuid
Apr 10, 2020
06b85eb
Merge pull request #178 from testcontainers/fix/uuid-gofrs
gianarb Apr 10, 2020
d0e9b43
Added ContainerIP to retrieve ip of container from primary network
gavinbunney Nov 4, 2019
042661a
Merge pull request #179 from testcontainers/feature/containerIP
gianarb Apr 14, 2020
00810a7
chore: ignore .idea
e-zhydzetski Mar 3, 2020
a930d9b
fix: improve error check in the ForListeningPort waiting strategy, fi…
e-zhydzetski Mar 3, 2020
adb8f34
Revert "chore: ignore .idea"
e-zhydzetski Mar 4, 2020
cabddc2
Merge pull request #180 from testcontainers/fix/159-wait-port-windows
gianarb Apr 14, 2020
a18968f
Merge pull request #176 from ikolomiyets/issue-173
gianarb Apr 14, 2020
1276796
(#25) Add utility to generate random strings
mdelapenya Sep 11, 2019
790421c
(#25) Create very basic implementation of Local Docker Compose implem…
mdelapenya Sep 11, 2019
0ed0ea9
(#25) Add a function to "down" the compose
mdelapenya Sep 11, 2019
fe96445
(#25) Make ExecError public
mdelapenya Sep 11, 2019
b8299a7
(#25) Use lowercase for container/service identifiers
mdelapenya Sep 18, 2019
06ada3c
(#25) Fix compose down
mdelapenya Sep 18, 2019
a320e66
(#25) Add a test verifying that the container receives the env vars
mdelapenya Sep 18, 2019
1badb29
(#25) Extract assertion of container env to a function
mdelapenya Sep 18, 2019
47362c1
(#25) Extract default environment generation to a function
mdelapenya Sep 18, 2019
d9e1579
(#25) Extract compose execution to a function
mdelapenya Sep 18, 2019
2604294
(#25) Support passing multiple docker-compose files
mdelapenya Sep 18, 2019
4807a91
(#25) Support checking for absent env vars in tests
mdelapenya Sep 18, 2019
80f3758
(#25) Make tests more readable
mdelapenya Sep 18, 2019
7614627
(#25) Remove unneeded log
mdelapenya Oct 29, 2019
28b4ce4
(#25) Use Google's UUID for generating compose's identifier
mdelapenya Nov 18, 2019
866cd2a
(#25) Add local compose to the docs
mdelapenya Nov 18, 2019
26eee98
(#25) Rename simple compose test resource
mdelapenya Nov 18, 2019
8096d05
(#25) Add a test with a more complex scenario
mdelapenya Nov 18, 2019
efa9db2
(#25) Fix path in test
mdelapenya Nov 18, 2019
3c26102
(#25) Add info about the command and the arguments when compose fails
mdelapenya Nov 19, 2019
a07447a
(#25) Create defer functions the soonest
mdelapenya Nov 20, 2019
a3e1f0b
(#25) Add comments and Go Examples for docs
mdelapenya Nov 22, 2019
ef1c203
(#25) Do not override ports on Travis CI
mdelapenya Nov 26, 2019
71af051
(#25) Support getting the services from the compose files
mdelapenya Sep 30, 2019
956a03b
(#25) Execute go mod tidy
mdelapenya Nov 26, 2019
446944d
(#25) Provide more information when failing a Compose test
mdelapenya Nov 26, 2019
b587960
(#25) Extract array population on exec errors
mdelapenya Nov 26, 2019
661033b
(#25) Enhance exec error
mdelapenya Nov 26, 2019
88efbd0
(#25) Optimise reading for docker-compose output
mdelapenya Nov 27, 2019
9570dd9
(#25) Do not assert the environment variables in the simple test
mdelapenya Nov 27, 2019
7182ddc
(#25) Fix tests reading environment variables from the compose file
mdelapenya Dec 26, 2019
6830723
(#25) Add troubleshooting Travis guide
mdelapenya Dec 26, 2019
03c0820
(#25) Run go mod tidy
mdelapenya Apr 16, 2020
a911c8e
Merge pull request #97 from mdelapenya/25-docker-compose
gianarb Apr 20, 2020
5234e4d
docs: add example for wait http
Apr 20, 2020
5e4c7c3
Merge pull request #184 from testcontainers/fix/documentation_wait_http
gianarb Apr 20, 2020
8565be5
Bump github.com/stretchr/testify from 1.4.0 to 1.5.1
dependabot-preview[bot] Apr 21, 2020
ae2f962
chore: do not raise panic, use exec error instead so the client code …
mdelapenya Apr 22, 2020
1b553a7
Merge pull request #189 from mdelapenya/docker-compose-no-panic
gianarb Apr 23, 2020
dd76d1e
Merge pull request #187 from testcontainers/dependabot/go_modules/git…
gianarb Apr 23, 2020
5698660
Bump github.com/gin-gonic/gin from 1.6.2 to 1.6.3
dependabot-preview[bot] May 4, 2020
9e67f94
Merge pull request #190 from testcontainers/dependabot/go_modules/git…
gianarb May 5, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
68 changes: 68 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,3 +158,71 @@ if err != nil {
// do something with err
}
```

## Using Docker Compose

Similar to generic containers support, it's also possible to run a bespoke set of services specified in a docker-compose.yml file.

This is intended to be useful on projects where Docker Compose is already used in dev or other environments to define services that an application may be dependent upon.

You can override Testcontainers' default behaviour and make it use a docker-compose binary installed on the local machine. This will generally yield an experience that is closer to running docker-compose locally, with the caveat that Docker Compose needs to be present on dev and CI machines.

### Examples

```go
composeFilePaths := "testresources/docker-compose.yml"
identifier := strings.ToLower(uuid.New().String())

compose := tc.NewLocalDockerCompose(composeFilePaths, identifier)
execError := compose.
WithCommand([]string{"up", "-d"}).
WithEnv(map[string]string {
"key1": "value1",
"key2": "value2",
}).
Invoke()
err := execError.Error
if err != nil {
return fmt.Errorf("Could not run compose file: %v - %v", composeFilePaths, err)
}
return nil
```

Note that the environment variables in the `env` map will be applied, if possible, to the existing variables declared in the docker compose file.

In the following example, we demonstrate how to stop a Docker compose using the convenient `Down` method.

```go
composeFilePaths := "testresources/docker-compose.yml"

compose := tc.NewLocalDockerCompose(composeFilePaths, identifierFromExistingRunningCompose)
execError := compose.Down()
err := execError.Error
if err != nil {
return fmt.Errorf("Could not run compose file: %v - %v", composeFilePaths, err)
}
return nil
```

## Troubleshooting Travis

If you want to reproduce a Travis build locally, please follow this instructions to spin up a Travis build agent locally:
```shell
export BUILDID="build-testcontainers"
export INSTANCE="travisci/ci-sardonyx:packer-1564753982-0c06deb6"
docker run --name $BUILDID -w /root/go/src/github.com/testcontainers/testcontainers-go -v /Users/mdelapenya/sourcecode/src/github.com/mdelapenya/testcontainers-go:/root/go/src/github.com/testcontainers/testcontainers-go -v /var/run/docker.sock:/var/run/docker.sock -dit $INSTANCE /sbin/init
```

Once the container has been created, enter it (`docker exec -ti $BUILDID bash`) and reproduce Travis steps:

```shell
eval "$(gimme 1.11.4)"
export GO111MODULE=on
export GOPATH="/root/go"
export PATH="$GOPATH/bin:$PATH"
go get gotest.tools/gotestsum
go mod tidy
go fmt ./...
go vet ./...
gotestsum --format short-verbose ./...
```
269 changes: 269 additions & 0 deletions compose.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,269 @@
package testcontainers

import (
"bytes"
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
"sync"

"gopkg.in/yaml.v2"
)

const (
envProjectName = "COMPOSE_PROJECT_NAME"
envComposeFile = "COMPOSE_FILE"
)

// DockerCompose defines the contract for running Docker Compose
type DockerCompose interface {
Down() ExecError
Invoke() ExecError
WithCommand([]string) DockerCompose
WithEnv(map[string]string) DockerCompose
}

// LocalDockerCompose represents a Docker Compose execution using local binary
// docker-compose or docker-compose.exe, depending on the underlying platform
type LocalDockerCompose struct {
Executable string
ComposeFilePaths []string
absComposeFilePaths []string
Identifier string
Cmd []string
Env map[string]string
Services map[string]interface{}
}

// NewLocalDockerCompose returns an instance of the local Docker Compose, using an
// array of Docker Compose file paths and an identifier for the Compose execution.
//
// It will iterate through the array adding '-f compose-file-path' flags to the local
// Docker Compose execution. The identifier represents the name of the execution,
// which will define the name of the underlying Docker network and the name of the
// running Compose services.
func NewLocalDockerCompose(filePaths []string, identifier string) *LocalDockerCompose {
dc := &LocalDockerCompose{}

dc.Executable = "docker-compose"
if runtime.GOOS == "windows" {
dc.Executable = "docker-compose.exe"
}

dc.ComposeFilePaths = filePaths

dc.absComposeFilePaths = make([]string, len(filePaths))
for i, cfp := range dc.ComposeFilePaths {
abs, _ := filepath.Abs(cfp)
dc.absComposeFilePaths[i] = abs
}

dc.validate()

dc.Identifier = strings.ToLower(identifier)

return dc
}

// Down executes docker-compose down
func (dc *LocalDockerCompose) Down() ExecError {
return executeCompose(dc, []string{"down"})
}

func (dc *LocalDockerCompose) getDockerComposeEnvironment() map[string]string {
environment := map[string]string{}

composeFileEnvVariableValue := ""
for _, abs := range dc.absComposeFilePaths {
composeFileEnvVariableValue += abs + string(os.PathListSeparator)
}

environment[envProjectName] = dc.Identifier
environment[envComposeFile] = composeFileEnvVariableValue

return environment
}

// Invoke invokes the docker compose
func (dc *LocalDockerCompose) Invoke() ExecError {
return executeCompose(dc, dc.Cmd)
}

// WithCommand assigns the command
func (dc *LocalDockerCompose) WithCommand(cmd []string) DockerCompose {
dc.Cmd = cmd
return dc
}

// WithEnv assigns the environment
func (dc *LocalDockerCompose) WithEnv(env map[string]string) DockerCompose {
dc.Env = env
return dc
}

// validate checks if the files to be run in the compose are valid YAML files, setting up
// references to all services in them
func (dc *LocalDockerCompose) validate() error {
type compose struct {
Services map[string]interface{}
}

for _, abs := range dc.absComposeFilePaths {
c := compose{}

yamlFile, err := ioutil.ReadFile(abs)
if err != nil {
return err
}
err = yaml.Unmarshal(yamlFile, &c)
if err != nil {
return err
}

dc.Services = c.Services
}

return nil
}

// ExecError is super struct that holds any information about an execution error, so the client code
// can handle the result
type ExecError struct {
Command []string
Error error
Stdout error
Stderr error
}

// execute executes a program with arguments and environment variables inside a specific directory
func execute(
dirContext string, environment map[string]string, binary string, args []string) ExecError {

var errStdout, errStderr error

cmd := exec.Command(binary, args...)
cmd.Dir = dirContext
cmd.Env = os.Environ()

for key, value := range environment {
cmd.Env = append(cmd.Env, key+"="+value)
}

stdoutIn, _ := cmd.StdoutPipe()
stderrIn, _ := cmd.StderrPipe()

stdout := newCapturingPassThroughWriter(os.Stdout)
stderr := newCapturingPassThroughWriter(os.Stderr)

err := cmd.Start()
if err != nil {
execCmd := []string{"Starting command", dirContext, binary}
execCmd = append(execCmd, args...)

return ExecError{
// add information about the CMD and arguments used
Command: execCmd,
Error: err,
Stderr: errStderr,
Stdout: errStdout,
}
}

var wg sync.WaitGroup
wg.Add(1)

go func() {
_, errStdout = io.Copy(stdout, stdoutIn)
wg.Done()
}()

_, errStderr = io.Copy(stderr, stderrIn)
wg.Wait()

err = cmd.Wait()

execCmd := []string{"Reading std", dirContext, binary}
execCmd = append(execCmd, args...)

return ExecError{
Command: execCmd,
Error: err,
Stderr: errStderr,
Stdout: errStdout,
}
}

func executeCompose(dc *LocalDockerCompose, args []string) ExecError {
if which(dc.Executable) != nil {
return ExecError{
Command: []string{dc.Executable},
Error: fmt.Errorf("Local Docker Compose not found. Is %s on the PATH?", dc.Executable),
}
}

environment := dc.getDockerComposeEnvironment()
for k, v := range dc.Env {
environment[k] = v
}

cmds := []string{}
pwd := "."
if len(dc.absComposeFilePaths) > 0 {
pwd, _ = filepath.Split(dc.absComposeFilePaths[0])

for _, abs := range dc.absComposeFilePaths {
cmds = append(cmds, "-f", abs)
}
} else {
cmds = append(cmds, "-f", "docker-compose.yml")
}
cmds = append(cmds, args...)

execErr := execute(pwd, environment, dc.Executable, cmds)
err := execErr.Error
if err != nil {
args := strings.Join(dc.Cmd, " ")
return ExecError{
Command: []string{dc.Executable},
Error: fmt.Errorf("Local Docker compose exited abnormally whilst running %s: [%v]. %s", dc.Executable, args, err.Error()),
}
}

return execErr
}

// capturingPassThroughWriter is a writer that remembers
// data written to it and passes it to w
type capturingPassThroughWriter struct {
buf bytes.Buffer
w io.Writer
}

// newCapturingPassThroughWriter creates new capturingPassThroughWriter
func newCapturingPassThroughWriter(w io.Writer) *capturingPassThroughWriter {
return &capturingPassThroughWriter{
w: w,
}
}

func (w *capturingPassThroughWriter) Write(d []byte) (int, error) {
w.buf.Write(d)
return w.w.Write(d)
}

// Bytes returns bytes written to the writer
func (w *capturingPassThroughWriter) Bytes() []byte {
return w.buf.Bytes()
}

// Which checks if a binary is present in PATH
func which(binary string) error {
_, err := exec.LookPath(binary)

return err
}