fix: updatecli compose caching

[olblak]

Fix #1867

Well the first iteration was plainly wrong.
This pullrequest also checks that all files needed are downloaded and not only the root directory

Test

Test locally that

1. Running multiple time only pull the policy once
2. Deleting file in the policy root directory will trigger a new pull during the next run

Additional Information

Tradeoff

Potential improvement

[mcwarman]

Testing:

First run - downloads as expected:

~/code/src/github.com/updatecli/updatecli/bin/updatecli compose diff --experimental --debug
Experimental Mode Enabled
DEBUG: Initializing policy "TFLint Ruleset - AzureRM"
DEBUG: latest tag identified "0.3.0"
DEBUG: /tmp/updatecli/store/3bd6a75b9a075dbf4f26d8678dfc549b6942be664721ec2bbf57c7895f5c1e7d/updatecli.d/manifest.yaml does not exist locally
Pulling Updatecli policy "ghcr.io/.../...updatecli-policies/tflint-ruleset-azure:latest"
DEBUG: Manifests:
DEBUG:  *"/tmp/updatecli/store/3bd6a75b9a075dbf4f26d8678dfc549b6942be664721ec2bbf57c7895f5c1e7d/updatecli.d/manifest.yaml"
DEBUG: policy successfully pulled in /tmp/updatecli/store/3bd6a75b9a075dbf4f26d8678dfc549b6942be664721ec2bbf57c7895f5c1e7d
DEBUG: Initializing policy "pre-commit Terraform"
DEBUG: latest tag identified "0.4.0"
DEBUG: /tmp/updatecli/store/fef60de49b69fb95c8f855fd95bbf8205e60ac06cf4f0012ca4362ec1c5cb289/updatecli.d/manifest.yaml does not exist locally
Pulling Updatecli policy "ghcr.io/.../...updatecli-policies/pre-commit-terraform:latest"
DEBUG: Manifests:
DEBUG:  *"/tmp/updatecli/store/fef60de49b69fb95c8f855fd95bbf8205e60ac06cf4f0012ca4362ec1c5cb289/updatecli.d/manifest.yaml"
DEBUG: policy successfully pulled in /tmp/updatecli/store/fef60de49b69fb95c8f855fd95bbf8205e60ac06cf4f0012ca4362ec1c5cb289
DEBUG: Initializing policy "Terraform Provider - AzureRM"
DEBUG: latest tag identified "0.4.0"
DEBUG: /tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5/updatecli.d/manifest.yaml does not exist locally
DEBUG: /tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5/values.yaml does not exist locally
Pulling Updatecli policy "ghcr.io/.../...updatecli-policies/terraform-provider-azurerm:latest"
DEBUG: Manifests:
DEBUG:  *"/tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5/updatecli.d/manifest.yaml"
DEBUG: Values:
DEBUG:  *"/tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5/values.yaml"
DEBUG: policy successfully pulled in /tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5


+++++++++++
+ PREPARE +
+++++++++++

Loading Pipeline "/tmp/updatecli/store/3bd6a75b9a075dbf4f26d8678dfc549b6942be664721ec2bbf57c7895f5c1e7d/updatecli.d/manifest.yaml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: using pipeline name to generate the pipelineid
WARNING: Updatecli binary version is unset. This means you are using a development version that ignores manifest version constraint.
Loading Pipeline "/tmp/updatecli/store/fef60de49b69fb95c8f855fd95bbf8205e60ac06cf4f0012ca4362ec1c5cb289/updatecli.d/manifest.yaml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: using pipeline name to generate the pipelineid
Loading Pipeline "/tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5/updatecli.d/manifest.yaml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: using pipeline name to generate the pipelineid

Second run- reuses as expected:

~/code/src/github.com/updatecli/updatecli/bin/updatecli compose diff --experimental --debug
Experimental Mode Enabled
DEBUG: Initializing policy "TFLint Ruleset - AzureRM"
DEBUG: latest tag identified "0.3.0"
DEBUG: Policy "ghcr.io/.../...updatecli-policies/tflint-ruleset-azure:latest" already available in:
        * /tmp/updatecli/store/3bd6a75b9a075dbf4f26d8678dfc549b6942be664721ec2bbf57c7895f5c1e7d
DEBUG: Manifests:
DEBUG:  *"/tmp/updatecli/store/3bd6a75b9a075dbf4f26d8678dfc549b6942be664721ec2bbf57c7895f5c1e7d/updatecli.d/manifest.yaml"
DEBUG: policy successfully pulled in /tmp/updatecli/store/3bd6a75b9a075dbf4f26d8678dfc549b6942be664721ec2bbf57c7895f5c1e7d
DEBUG: Initializing policy "pre-commit Terraform"
DEBUG: latest tag identified "0.4.0"
DEBUG: Policy "ghcr.io/.../...updatecli-policies/pre-commit-terraform:latest" already available in:
        * /tmp/updatecli/store/fef60de49b69fb95c8f855fd95bbf8205e60ac06cf4f0012ca4362ec1c5cb289
DEBUG: Manifests:
DEBUG:  *"/tmp/updatecli/store/fef60de49b69fb95c8f855fd95bbf8205e60ac06cf4f0012ca4362ec1c5cb289/updatecli.d/manifest.yaml"
DEBUG: policy successfully pulled in /tmp/updatecli/store/fef60de49b69fb95c8f855fd95bbf8205e60ac06cf4f0012ca4362ec1c5cb289
DEBUG: Initializing policy "Terraform Provider - AzureRM"
DEBUG: latest tag identified "0.4.0"
DEBUG: Policy "ghcr.io/.../...updatecli-policies/terraform-provider-azurerm:latest" already available in:
        * /tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5
DEBUG: Manifests:
DEBUG:  *"/tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5/updatecli.d/manifest.yaml"
DEBUG: Values:
DEBUG:  *"/tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5/values.yaml"
DEBUG: policy successfully pulled in /tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5


+++++++++++
+ PREPARE +
+++++++++++

Loading Pipeline "/tmp/updatecli/store/3bd6a75b9a075dbf4f26d8678dfc549b6942be664721ec2bbf57c7895f5c1e7d/updatecli.d/manifest.yaml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: using pipeline name to generate the pipelineid
WARNING: Updatecli binary version is unset. This means you are using a development version that ignores manifest version constraint.
Loading Pipeline "/tmp/updatecli/store/fef60de49b69fb95c8f855fd95bbf8205e60ac06cf4f0012ca4362ec1c5cb289/updatecli.d/manifest.yaml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: using pipeline name to generate the pipelineid
Loading Pipeline "/tmp/updatecli/store/299bc812694dcbd23d9ecf1e47ccbb8fb2a03bfe4f92dc95a9e7be37e835dca5/updatecli.d/manifest.yaml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: using pipeline name to generate the pipelineid