Dependency var_files (#73)

* [skip ci] WIP: fix newline issues in examples and fixtures

* Implement VarFiles for dependencies

* Add support to report the junit results

* Fix broken tests

.circleci/config.yml

@@ -8,7 +8,7 @@ defaults: &defaults
     PACKER_VERSION: NONE
     GOLANG_VERSION: "1.14"
 
-version: 2
+version: 2.1
 jobs:
   test:
     <<: *defaults
@@ -26,11 +26,26 @@ jobs:
               --terragrunt-version ${TERRAGRUNT_VERSION} \
               --packer-version ${PACKER_VERSION} \
               --go-version ${GOLANG_VERSION}
+
+            go get -u github.com/jstemmer/go-junit-report
       - run:
           name: run tests
-          command: run-go-tests --path . --timeout 1h
+          command: |
+            mkdir -p /tmp/logs/results
+            run-go-tests --path . --timeout 1h | tee /tmp/logs/all.log
           no_output_timeout: 3600s
 
+
+      # Store junit test result so it shows up in the CircleCI ui
+      - run:
+          # Use go-junit-report to generate a junit report instead of terratest log parser, as the logs for boilerplate
+          # do not respect the logging mechanism that the log parser looks for.
+          name: parse logs
+          command: "cat /tmp/logs/all.log | go-junit-report > /tmp/logs/results/results.xml"
+          when: always
+      - store_test_results:
+          path: /tmp/logs
+
   deploy:
     <<: *defaults
     steps:

README.md

@@ -342,6 +342,15 @@ executing the current one. Each dependency may contain the following keys:
   but you want the dependency to get a different value for the variable, you can specify overrides here. `boilerplate`
   will include a separate prompt for variables defined under a `dependency`. You can also override the dependency's
   description and default values here.
+* `var_files`: If you want to set the variables for the dependency based on a var file, you can provide a list of var
+  file paths to include. The path is assumed to be relative to the current boilerplate.yml, but it is recommended to use
+  the `templateFolder` helper function to be explicit. Note that the order of preference for variables are (top most
+  have highest precedence - aka override all):
+    - Top level variables set on the CLI.
+    - Var files set on the dependency.
+    - Defaults set on dependency variables (the `variables` field of the dependency).
+    - Defaults set on root variables.
+    - Defaults set within the dependency boilerplate config.
 
 See the [Dependencies](#dependencies) section for more info.
 

examples/dependencies-varfile-precedence/README.md

@@ -0,0 +1,6 @@
+# {{ .Title }}
+
+{{ .Description }}. It specifies both the
+[docs](/examples/docs) and [website](/examples/website) examples as dependencies to show how one boilerplate template
+can pull in another. It also defines all the variables needed for both of those dependencies at the top level to show
+how variable inheritance works.

examples/dependencies-varfile-precedence/boilerplate.yml

@@ -0,0 +1,33 @@
+variables:
+  - name: Description
+    description: Enter the description of this template
+
+  - name: Version
+    description: Enter the version number that will be used by the docs dependency
+
+  - name: Title
+    description: Enter the title for the dependencies example
+
+  - name: WelcomeText
+    description: Enter the welcome text used by the website dependency
+
+  - name: ShowLogo
+    description: Should the webiste show the logo (true or false)?
+    type: bool
+    default: true
+
+dependencies:
+  - name: docs
+    template-url: ../docs
+    output-folder: ./docs
+    var_files:
+      - "{{ templateFolder }}/docs_vars.yml"
+
+  - name: website
+    template-url: ../website
+    output-folder: ./website
+    var_files:
+      - "{{ templateFolder }}/website_vars.yml"
+
+skip_files:
+  - path: "*_vars.yml"

examples/dependencies-varfile-precedence/docs_vars.yml

@@ -0,0 +1 @@
+Title: "I am vars for docs"

examples/dependencies-varfile-precedence/website_vars.yml

@@ -0,0 +1 @@
+Title: "website vars!"

examples/dependencies-varfile/README.md

@@ -0,0 +1,6 @@
+# {{ .Title }}
+
+{{ .Description }}. It specifies both the
+[docs](/examples/docs) and [website](/examples/website) examples as dependencies to show how one boilerplate template
+can pull in another. It also defines all the variables needed for both of those dependencies at the top level to show
+how variable inheritance works.

examples/dependencies-varfile/boilerplate.yml

@@ -0,0 +1,33 @@
+variables:
+  - name: Description
+    description: Enter the description of this template
+
+  - name: Version
+    description: Enter the version number that will be used by the docs dependency
+
+  - name: Title
+    description: Enter the title for the dependencies example
+
+  - name: WelcomeText
+    description: Enter the welcome text used by the website dependency
+
+  - name: ShowLogo
+    description: Should the webiste show the logo (true or false)?
+    type: bool
+    default: true
+
+dependencies:
+  - name: docs
+    template-url: ../docs
+    output-folder: ./docs
+    var_files:
+      - "{{ templateFolder }}/docs_vars.yml"
+
+  - name: website
+    template-url: ../website
+    output-folder: ./website
+    var_files:
+      - "{{ templateFolder }}/website_vars.yml"
+
+skip_files:
+  - path: "*_vars.yml"

examples/dependencies-varfile/docs_vars.yml

@@ -0,0 +1 @@
+Title: "I am vars for docs"

examples/dependencies-varfile/website_vars.yml

@@ -0,0 +1 @@
+Title: "website vars!"

examples/docs/README.md

@@ -16,7 +16,7 @@ passed in using the `--var` option.
 Here is how to use the `snippet` helper to embed files or parts of files from source code:
 
 ```html
-{{snippet "../website/index.html"}}
+{{snippet "../website/index.html" | trim}}
 ```
 
 ## Arithmetic
@@ -34,4 +34,3 @@ And here is another way to do it using the slice helper:
 {{ range $i := (slice 1 4 1) -}}
 {{$i}}. Item
 {{ end -}}
-

examples/website/index.html

@@ -6,4 +6,4 @@
     <h1>{{.WelcomeText}}</h1>
     {{if .ShowLogo}}<img src="logo.png">{{end}}
   </body>
-</html>
+</html>

templates/template_processor.go

@@ -231,12 +231,26 @@ func cloneOptionsForDependency(dependency variables.Dependency, originalOpts *op
 	// Output folder should be local path relative to original output folder, or absolute path
 	outputFolder := render.PathRelativeToTemplate(originalOpts.OutputFolder, renderedOutputFolder)
 
+	renderedVarFiles := []string{}
+	for _, varFilePath := range dependency.VarFiles {
+		renderedVarFilePath, err := render.RenderTemplateFromString(originalOpts.TemplateFolder, varFilePath, variables, originalOpts)
+		if err != nil {
+			return nil, err
+		}
+		renderedVarFiles = append(renderedVarFiles, renderedVarFilePath)
+	}
+
+	vars, err := cloneVariablesForDependency(dependency, variables, renderedVarFiles)
+	if err != nil {
+		return nil, err
+	}
+
 	return &options.BoilerplateOptions{
 		TemplateUrl:     templateUrl,
 		TemplateFolder:  templateFolder,
 		OutputFolder:    outputFolder,
 		NonInteractive:  originalOpts.NonInteractive,
-		Vars:            cloneVariablesForDependency(dependency, variables),
+		Vars:            vars,
 		OnMissingKey:    originalOpts.OnMissingKey,
 		OnMissingConfig: originalOpts.OnMissingConfig,
 		DisableHooks:    originalOpts.DisableHooks,
@@ -246,12 +260,18 @@ func cloneOptionsForDependency(dependency variables.Dependency, originalOpts *op
 
 // Clone the given variables for use when rendering the given dependency.  The dependency will get the same variables
 // as the originals passed in, filtered to variable names that do not include a dependency or explicitly are for the
-// given dependency. If dependency.DontInheritVariables is set to true, an empty map is returned.
-func cloneVariablesForDependency(dependency variables.Dependency, originalVariables map[string]interface{}) map[string]interface{} {
-	newVariables := map[string]interface{}{}
+// given dependency.
+// If the dependency specifies VarFiles, set the initial variables based on each var file. Note that we prefer the
+// variables set on the CLI (originalVariables) over those set on the dependency (unless DontInheritVariables is set)
+// If dependency.DontInheritVariables is set to true, return just the variables set on the var files.
+func cloneVariablesForDependency(dependency variables.Dependency, originalVariables map[string]interface{}, renderedVarFiles []string) (map[string]interface{}, error) {
+	newVariables, err := variables.ParseVars(nil, renderedVarFiles)
+	if err != nil {
+		return nil, err
+	}
 
 	if dependency.DontInheritVariables {
-		return newVariables
+		return newVariables, nil
 	}
 
 	for variableName, variableValue := range originalVariables {
@@ -263,7 +283,7 @@ func cloneVariablesForDependency(dependency variables.Dependency, originalVariab
 		}
 	}
 
-	return newVariables
+	return newVariables, nil
 }
 
 // Prompt the user to verify if the given dependency should be executed and return true if they confirm. If

templates/template_processor_test.go

@@ -5,6 +5,7 @@ import (
 	"testing"
 
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 
 	"github.com/gruntwork-io/boilerplate/options"
 	"github.com/gruntwork-io/boilerplate/variables"
@@ -117,7 +118,8 @@ func TestCloneVariablesForDependency(t *testing.T) {
 	}
 
 	for _, testCase := range testCases {
-		actualVariables := cloneVariablesForDependency(testCase.dependency, testCase.variables)
+		actualVariables, err := cloneVariablesForDependency(testCase.dependency, testCase.variables, nil)
+		require.NoError(t, err)
 		assert.Equal(t, testCase.expectedVariables, actualVariables, "Dependency: %s", testCase.dependency)
 	}
 }

test-fixtures/examples-expected-output/dependencies-dynamic/website/index.html

@@ -6,4 +6,4 @@
     <h1>Welcome!</h1>
     <img src="logo.png">
   </body>
-</html>
+</html>

test-fixtures/examples-expected-output/dependencies-recursive/dependencies/website/index.html

@@ -6,4 +6,4 @@
     <h1>Welcome!</h1>
     <img src="logo.png">
   </body>
-</html>
+</html>

test-fixtures/examples-expected-output/dependencies-remote/website/index.html

@@ -6,4 +6,4 @@
     <h1>Welcome!</h1>
     <img src="logo.png">
   </body>
-</html>
+</html>

test-fixtures/examples-expected-output/dependencies-varfile-precedence/README.md

@@ -0,0 +1,6 @@
+# Dependencies example
+
+This is a boilerplate template that shows an example of using dependencies. It specifies both the
+[docs](/examples/docs) and [website](/examples/website) examples as dependencies to show how one boilerplate template
+can pull in another. It also defines all the variables needed for both of those dependencies at the top level to show
+how variable inheritance works.

test-fixtures/examples-expected-output/dependencies-varfile-precedence/docs/README.md

@@ -0,0 +1,42 @@
+# Docs example
+
+This shows an example of how to use boilerplate to fill in parts of your documentation.
+
+## Variables
+
+Here is how you can use a variable:
+
+The latest version of my app is 0.0.3.
+
+You could create a CI job that, for each release, regenerates your docs with the latest value of the `Version` variable
+passed in using the `--var` option.
+
+## Snippets
+
+Here is how to use the `snippet` helper to embed files or parts of files from source code:
+
+```html
+<html>
+  <head>
+    <title>{{.Title}}</title>
+  </head>
+  <body>
+    <h1>{{.WelcomeText}}</h1>
+    {{if .ShowLogo}}<img src="logo.png">{{end}}
+  </body>
+</html>
+```
+
+## Arithmetic
+
+Here is how you can use the arithmetic helpers to create a numbered list:
+
+1. Item
+2. Item
+3. Item
+
+And here is another way to do it using the slice helper:
+
+1. Item
+2. Item
+3. Item

test-fixtures/examples-expected-output/dependencies-varfile-precedence/docs/interpolated-folder-example-folder/my_example_file.py

@@ -0,0 +1 @@
+# This file and its parent folder both show an example of using Go template syntax and boilerplate variables in their names

test-fixtures/examples-expected-output/dependencies-varfile-precedence/website/index.html

@@ -0,0 +1,9 @@
+<html>
+  <head>
+    <title>Boilerplate</title>
+  </head>
+  <body>
+    <h1>Welcome!</h1>
+    <img src="logo.png">
+  </body>
+</html>

test-fixtures/examples-expected-output/dependencies-varfile/README.md

@@ -0,0 +1,6 @@
+# Dependencies example
+
+This is a boilerplate template that shows an example of using dependencies. It specifies both the
+[docs](/examples/docs) and [website](/examples/website) examples as dependencies to show how one boilerplate template
+can pull in another. It also defines all the variables needed for both of those dependencies at the top level to show
+how variable inheritance works.

test-fixtures/examples-expected-output/dependencies-varfile/docs/README.md

@@ -0,0 +1,42 @@
+# I am vars for docs
+
+This shows an example of how to use boilerplate to fill in parts of your documentation.
+
+## Variables
+
+Here is how you can use a variable:
+
+The latest version of my app is 0.0.3.
+
+You could create a CI job that, for each release, regenerates your docs with the latest value of the `Version` variable
+passed in using the `--var` option.
+
+## Snippets
+
+Here is how to use the `snippet` helper to embed files or parts of files from source code:
+
+```html
+<html>
+  <head>
+    <title>{{.Title}}</title>
+  </head>
+  <body>
+    <h1>{{.WelcomeText}}</h1>
+    {{if .ShowLogo}}<img src="logo.png">{{end}}
+  </body>
+</html>
+```
+
+## Arithmetic
+
+Here is how you can use the arithmetic helpers to create a numbered list:
+
+1. Item
+2. Item
+3. Item
+
+And here is another way to do it using the slice helper:
+
+1. Item
+2. Item
+3. Item

test-fixtures/examples-expected-output/dependencies-varfile/docs/interpolated-folder-example-folder/my_example_file.py

@@ -0,0 +1 @@
+# This file and its parent folder both show an example of using Go template syntax and boilerplate variables in their names

test-fixtures/examples-expected-output/dependencies-varfile/website/index.html

@@ -0,0 +1,9 @@
+<html>
+  <head>
+    <title>website vars!</title>
+  </head>
+  <body>
+    <h1>Welcome!</h1>
+    <img src="logo.png">
+  </body>
+</html>

test-fixtures/examples-expected-output/dependencies/website/index.html

@@ -6,4 +6,4 @@
     <h1>Welcome!</h1>
     <img src="logo.png">
   </body>
-</html>
+</html>