Add docker compose file autodiscovery

[olblak]

Signed-off-by: Olblak me@olblak.com

Fix #975

Add autodiscovery for docker compose file matching

• docker-compose.yaml
• docker-compose.*.yaml
• docker-compose.yml
• docker-compose.*.yml

Test

To test this pull request, you can run the following commands:

# Run UT
cd pkg/plugins/autodiscovery/dockercompose/
go test
## Test with existing Docker compose file

go build -o /tmp/updateCli
cd <to  a directory containing a dockerc-compose.yaml>
/tmp/updateCli diff --experimental --local-autodiscovery

Example

The autodiscovery can be used with and without manifest.
Without manifest you can just execute ./bin/updatecli diff --experimental

Manifest

./bin/updatecli diff --experimental --config manifest.yaml

.manifest.yaml

name: "Docker compose autodiscovery using git scm"
scms:
  default:
    kind: git
    spec:
      url: https://github.com/updatecli/updatemonitor.git

autodiscovery:
  # scmid is applied to all crawlers
  scmid: default
  crawlers:
    dockercompose:
      ignore: 
       - path: "*/docker-compose.yaml"
         services:
          -  traefik
      #only:
      #  # - path: <filepath relative to scm repository>
      #  # - path: chart/*

Console Output

Experimental Mode Enabled


+++++++++++
+ PREPARE +
+++++++++++


SCM repository retrieved: 0


++++++++++++++++++
+ AUTO DISCOVERY +
++++++++++++++++++



#######################
# LOCAL AUTODISCOVERY #
#######################


HELM
=====
Manifest detected: 0


RANCHER FLEET
==============
Manifest detected: 0


MAVEN
======
Manifest detected: 0


DOCKER COMPOSE
===============
WARNING: no credentials found for docker registry "index.docker.io" hosting image "mongo"
WARNING: no credentials found for docker registry "ghcr.io" hosting image "ghcr.io/updatecli/updatemonitor"
WARNING: no credentials found for docker registry "ghcr.io" hosting image "ghcr.io/updatecli/updatemonitor"
WARNING: no credentials found for docker registry "ghcr.io" hosting image "ghcr.io/updatecli/updatemonitor-ui"
WARNING: no credentials found for docker registry "index.docker.io" hosting image "traefik"
Manifest detected: 5


---

=> Total manifest detected: 5

WARNING: Autodiscovery settings "groupby" is undefined, fallback to "individual"


++++++++++++
+ PIPELINE +
++++++++++++



#######################
# LOCAL AUTODISCOVERY #
#######################


########################################################################################
# BUMP "MONGO" DOCKER COMPOSE SERVICE IMAGE VERSION FOR "TESTDATA/DOCKER-COMPOSE.YAML" #
########################################################################################


SOURCES
=======

mongodb
-------
Searching for version matching pattern "*"
✔ Docker Image Tag "6.0.2" found matching pattern "*"


TARGETS
========

mongodb
-------

**Dry Run enabled**

✔ Key 'services.mongodb.image', from file '/home/olblak/Projects/Updatecli/updatecli/pkg/plugins/autodiscovery/dockercompose/testdata/docker-compose.yaml', already set to mongo:6.0.2, nothing else need to be done


##################################################################################################################
# BUMP "GHCR.IO/UPDATECLI/UPDATEMONITOR" DOCKER COMPOSE SERVICE IMAGE VERSION FOR "TESTDATA/DOCKER-COMPOSE.YAML" #
##################################################################################################################


SOURCES
=======

server
------
Searching for version matching pattern "*"
✔ Docker Image Tag "v0.1.0" found matching pattern "*"


TARGETS
========

server
------

**Dry Run enabled**

✔ Key 'services.server.image', from file '/home/olblak/Projects/Updatecli/updatecli/pkg/plugins/autodiscovery/dockercompose/testdata/docker-compose.yaml', already set to ghcr.io/updatecli/updatemonitor:v0.1.0, nothing else need to be done


##################################################################################################################
# BUMP "GHCR.IO/UPDATECLI/UPDATEMONITOR" DOCKER COMPOSE SERVICE IMAGE VERSION FOR "TESTDATA/DOCKER-COMPOSE.YAML" #
##################################################################################################################


SOURCES
=======

agent
-----
Searching for version matching pattern "*"
✔ Docker Image Tag "v0.1.0" found matching pattern "*"


TARGETS
========

agent
-----

**Dry Run enabled**

✔ Key 'services.agent.image', from file '/home/olblak/Projects/Updatecli/updatecli/pkg/plugins/autodiscovery/dockercompose/testdata/docker-compose.yaml', already set to ghcr.io/updatecli/updatemonitor:v0.1.0, nothing else need to be done


#####################################################################################################################
# BUMP "GHCR.IO/UPDATECLI/UPDATEMONITOR-UI" DOCKER COMPOSE SERVICE IMAGE VERSION FOR "TESTDATA/DOCKER-COMPOSE.YAML" #
#####################################################################################################################


SOURCES
=======

front
-----
Searching for version matching pattern "*"
✔ Docker Image Tag "v0.1.1" found matching pattern "*"


TARGETS
========

front
-----

**Dry Run enabled**

✔ Key 'services.front.image', from file '/home/olblak/Projects/Updatecli/updatecli/pkg/plugins/autodiscovery/dockercompose/testdata/docker-compose.yaml', already set to ghcr.io/updatecli/updatemonitor-ui:v0.1.1, nothing else need to be done


##########################################################################################
# BUMP "TRAEFIK" DOCKER COMPOSE SERVICE IMAGE VERSION FOR "TESTDATA/DOCKER-COMPOSE.YAML" #
##########################################################################################


SOURCES
=======

traefik
-------
Searching for version matching pattern "*"
✔ Docker Image Tag "2.9.4" found matching pattern "*"


TARGETS
========

traefik
-------

**Dry Run enabled**

⚠ Key 'services.traefik.image', from file '/home/olblak/Projects/Updatecli/updatecli/pkg/plugins/autodiscovery/dockercompose/testdata/docker-compose.yaml', was updated from 'traefik:v2.9' to 'traefik:2.9.4'

=============================

REPORTS:


✔ Local AutoDiscovery:

✔ Bump "mongo" Docker compose service image version for "testdata/docker-compose.yaml":
	Source:
		✔ [mongodb] Get latest "mongo" Docker Image Tag (kind: dockerimage)
	Target:
		✔ [mongodb] Bump "mongo" Docker Image tag for docker compose file "testdata/docker-compose.yaml" (kind: yaml)


✔ Bump "ghcr.io/updatecli/updatemonitor" Docker compose service image version for "testdata/docker-compose.yaml":
	Source:
		✔ [server] Get latest "ghcr.io/updatecli/updatemonitor" Docker Image Tag (kind: dockerimage)
	Target:
		✔ [server] Bump "ghcr.io/updatecli/updatemonitor" Docker Image tag for docker compose file "testdata/docker-compose.yaml" (kind: yaml)


✔ Bump "ghcr.io/updatecli/updatemonitor" Docker compose service image version for "testdata/docker-compose.yaml":
	Source:
		✔ [agent] Get latest "ghcr.io/updatecli/updatemonitor" Docker Image Tag (kind: dockerimage)
	Target:
		✔ [agent] Bump "ghcr.io/updatecli/updatemonitor" Docker Image tag for docker compose file "testdata/docker-compose.yaml" (kind: yaml)


✔ Bump "ghcr.io/updatecli/updatemonitor-ui" Docker compose service image version for "testdata/docker-compose.yaml":
	Source:
		✔ [front] Get latest "ghcr.io/updatecli/updatemonitor-ui" Docker Image Tag (kind: dockerimage)
	Target:
		✔ [front] Bump "ghcr.io/updatecli/updatemonitor-ui" Docker Image tag for docker compose file "testdata/docker-compose.yaml" (kind: yaml)


⚠ Bump "traefik" Docker compose service image version for "testdata/docker-compose.yaml":
	Source:
		✔ [traefik] Get latest "traefik" Docker Image Tag (kind: dockerimage)
	Target:
		⚠ [traefik] Bump "traefik" Docker Image tag for docker compose file "testdata/docker-compose.yaml" (kind: yaml)


Run Summary
===========
Pipeline(s) run:
  * Changed:	1
  * Failed:	0
  * Skipped:	0
  * Succeeded:	5
  * Total:	6

Additional Information

Tradeoff

Potential improvement

• Remove credentials warning message, since we switch to google container registry, we can now use by default the local credential file
• Refactor both this plugin and the helm chart autodiscovery to implement DRY
• Guess versionFilter rule base on the tag
  ** 1.0.0 semver
  ** 1.0.0-linux (regex)
• Right now we assume that the docker image is linux. So improvement could also be done to detect the architecture

[olblak]

This pullrquest is ready to be merged.
I am planning to implement the smart guessing for versionFilter in a separated pullrequest as it's not specific to docker-compose

[olblak]

This pullrequest is feature ready but I still need to review it

[dduportal]

@olblak I just pushed a commit with my proposed changes. Feel free to remove it, edit it or whatever.

The goal of this commit is to fix the flakiness by changing the internal data struct to ordered on. It's sorted as soon as the found YAML file are found and unmarshalled to you can rely on the lexicographic order as soon as you get the custom data with your services attributes.

I haven't run the full e2e suite: I wanted to get the change to you as soon as possible.