This repository has been archived by the owner on Apr 17, 2023. It is now read-only.
Garbage Collector removes all tags #2241
Labels
Comments
|
This is affecting us very badly. any update on this? |
Hi @robgiovanardi ? I have a feeling, but no more time just wanted to give you an important tip :
Btw, Garbage collection sounds soooo much like a background job, doesn't it ? (Imagine a Hope this will help |
|
Hi @Jean-Baptiste-Lasselle Only portus_backgound container has So, yes, we are running garbage collection on Portus Background dedicated container and the problem is still here.. |
|
We also have the same problem and can confirm that the keep_latest setting doesn’t work. |
hi @robgiovanardi thank you so much for ur feedback, i mean it's very interesting, i haven't tested the feature yet, but it's at least very important business case. But i can try n help you :
|
|
@robgiovanardi all in all, since some images (all of them) are deleted, the communication should be ok, i think you've got your hands on a real bug inside I have another idea, too : for your |
I forgot to mention environment variable used for background and foreground, I'm adding that to Description |
Actually, yes, because we are using the very same config.yaml for both foreground and background, and running them with different environment variables, I just updated initial description including those envs. Let me try a different config.yml |
Excellent news @robgiovanardi !!!! Indeed, my idea was that the comunication between your registry and the
One final amusement remark : The examples in the portus distrib are awful..,And it's funny I heave a feeling thy were kind of teared off a legacy docker swarm cluster... I Might be wrong. Or not . :) |
yes, do your thing with the
having accurate results on that test, will help me a lot. Still is sure that we will have to ironize your network setup, so it's more explicitly telling the operator who 's talking to who and for what purpose. You security guys will like that too. |
Let me better understand: communication would happens from portus_background to registry, right? Not from registry to portus_background?
Yes, you right, the registry is installed as legacy application, with zypper, not deployed with docker nor docker-compose
|
|
Hi @Jean-Baptiste-Lasselle
The problem is still there Things to notice: I enabled debug mode on portus_background: it drop this query to find if there's some images to delete: I used Some debug logs with |
|
hi @robgiovanardi : Communication between
|
|
hi @robgiovanardi just to keep you informed, I am totally working on the matter, still haven't finished complete automation, and I today found something that sounds very good, in relation with garbage collection : #2275 (comment) What those # Sync config
- PORTUS_BACKGROUND_REGISTRY_ENABLED=true
- PORTUS_BACKGROUND_SYNC_ENABLED=true
- PORTUS_BACKGROUND_SYNC_STRATEGY=updatemade me think ... Maybe there exists things like
Never the less, I wouldn't be surprised that the |
|
@robgiovanardi I think I found your solution!!!!, and oh my god, the idea I just wrote 5minutes ago, inspred by #2275 (comment) , gave full reward !!!! have a look out there : https://github.com/Ashtonian/server-setup/blob/bc9ac031a18f1c686da5a662d3cf969009a50c38/portus/docker-compose.yml So Yesssss! there exist And so, what you need to do, is to ad the following env. variables to both your - PORTUS_DELETE_ENABLED=true
- PORTUS_DELETE_CONTRIBUTORS=false
- PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
- PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=30
- PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=5Honestly, I'll try as soon aspossible to set that only for the background, just to check if it works, cause there 's a potential non-necessary copy-paste in this example. I have a lot of other work on Portus, so I can't do that this weekend, I am so dying that you do that and give me feedback even before I run it 😄 The whole docker-compose.yml, so we don't lose itI found it seraching github with string Even funnier, 😆 , none of those 4 results are in portus documentation !! I had take the screenshot before there are more results on github.com ! 😆
version: "3.7"
services:
portus:
image: opensuse/portus:2.4.3
# env_file:
# - ./portus.env
environment:
- PORTUS_MACHINE_FQDN_VALUE=portus.ashlab.dev
- PORTUS_DB_HOST=db
- PORTUS_DB_DATABASE=portus_production
- PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
- PORTUS_DB_POOL=5
- PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
- PORTUS_KEY_PATH=/certificates/portus.ashlab.dev/privatekey.key
- PORTUS_PASSWORD=${PORTUS_PASSWORD}
- PORTUS_CHECK_SSL_USAGE_ENABLED=false
- PORTUS_SIGNUP_ENABLED=false
- RAILS_SERVE_STATIC_FILES=true
- PORTUS_GRAVATAR_ENABLED=true
- PORTUS_DELETE_ENABLED=true
- PORTUS_DELETE_CONTRIBUTORS=false
- PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
- PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=30
- PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=5
- PORTUS_ANONYMOUS_BROWSING_ENABLED=false
- PORTUS_OAUTH_GITHUB_ENABLED=true
- PORTUS_OAUTH_GITHUB_CLIENT_ID=${PORTUS_OAUTH_GITHUB_CLIENT_ID}
- PORTUS_OAUTH_GITHUB_CLIENT_SECRET=${PORTUS_OAUTH_GITHUB_CLIENT_SECRET}
- PORTUS_OAUTH_GITHUB_ORGANIZATION=karsto
# # - PORTUS_OAUTH_GITHUB_TEAM=''
# # - PORTUS_OAUTH_GITHUB_DOMAIN=''
# - PORTUS_SECURITY_CLAIR_SERVER=http://clair:6060
# ports:
# - 3000:3000
depends_on:
- db
links:
- db
volumes:
- traefik_certs_raw:/certificates:ro
# - secrets:/certificates:ro
networks:
- portus
- public
labels:
- "traefik.enable=true"
# - "traefik.http.middlewares.sslHeaders.headers.SSLHost=portus.ashlab.dev"
- "traefik.http.routers.portus.rule=Host(`portus.ashlab.dev`)"
- "traefik.http.routers.portus.middlewares=https_redirect, sslHeaders"
- "traefik.http.routers.portus.service=portus"
- "traefik.http.routers.portus.tls=true"
- "traefik.http.routers.portus.tls.certresolver=le"
- "traefik.http.services.portus.loadbalancer.server.port=3000"
- "traefik.http.services.portus.loadbalancer.server.scheme=http"
- "traefik.http.middlewares.https_redirect.redirectscheme.scheme=https" # Standard move to default when traefik fixes behavior
- "traefik.http.middlewares.https_redirect.redirectscheme.permanent=true"
# - "traefik.http.middlewares.sslHeaders.headers.framedeny=true"
# - "traefik.http.middlewares.sslHeaders.headers.sslredirect=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSSeconds=315360000"
# - "traefik.http.middlewares.sslHeaders.headers.browserXSSFilter=true"
# - "traefik.http.middlewares.sslHeaders.headers.contentTypeNosniff=true"
# - "traefik.http.middlewares.sslHeaders.headers.forceSTSHeader=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSIncludeSubdomains=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSPreload=true"
deploy:
labels:
- "traefik.enable=true"
# - "traefik.http.middlewares.sslHeaders.headers.SSLHost=portus.ashlab.dev"
- "traefik.http.routers.portus.rule=Host(`portus.ashlab.dev`)"
- "traefik.http.routers.portus.middlewares=https_redirect, sslHeaders"
- "traefik.http.routers.portus.service=portus"
- "traefik.http.routers.portus.tls=true"
- "traefik.http.routers.portus.tls.certresolver=le"
- "traefik.http.services.portus.loadbalancer.server.port=3000"
- "traefik.http.services.portus.loadbalancer.server.scheme=http"
# - "traefik.http.middlewares.https_redirect.redirectscheme.scheme=https" # Standard move to default when traefik fixes behavior
# - "traefik.http.middlewares.https_redirect.redirectscheme.permanent=true"
# - "traefik.http.middlewares.sslHeaders.headers.framedeny=true"
# - "traefik.http.middlewares.sslHeaders.headers.sslredirect=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSSeconds=315360000"
# - "traefik.http.middlewares.sslHeaders.headers.browserXSSFilter=true"
# - "traefik.http.middlewares.sslHeaders.headers.contentTypeNosniff=true"
# - "traefik.http.middlewares.sslHeaders.headers.forceSTSHeader=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSIncludeSubdomains=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSPreload=true"
background:
image: opensuse/portus:2.4.3
depends_on:
- portus
- db
environment:
# Theoretically not needed, but cconfig's been buggy on this...
- CCONFIG_PREFIX=PORTUS
- PORTUS_MACHINE_FQDN_VALUE=portus.ashlab.dev
- PORTUS_DB_HOST=db
- PORTUS_DB_DATABASE=portus_production
- PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
- PORTUS_DB_POOL=5
- PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
- PORTUS_KEY_PATH=/certificates/portus.ashlab.dev/privatekey.key
- PORTUS_PASSWORD=${PORTUS_PASSWORD}
# - PORTUS_SECURITY_CLAIR_SERVER=http://clair:6060
# - PORTUS_CHECK_SSL_USAGE_ENABLED=false
- PORTUS_GRAVATAR_ENABLED=true
- PORTUS_DELETE_ENABLED=true
- PORTUS_DELETE_CONTRIBUTORS=false
- PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
- PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=30
- PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=5
- PORTUS_OAUTH_GITHUB_ENABLED=true
- PORTUS_OAUTH_GITHUB_CLIENT_ID=${PORTUS_OAUTH_GITHUB_CLIENT_ID}
- PORTUS_OAUTH_GITHUB_CLIENT_SECRET=${PORTUS_OAUTH_GITHUB_CLIENT_SECRET}
- PORTUS_OAUTH_GITHUB_ORGANIZATION=karsto
# - PORTUS_OAUTH_GITHUB_TEAM=''
# - PORTUS_OAUTH_GITHUB_DOMAIN=''
- PORTUS_ANONYMOUS_BROWSING_ENABLED=false
- PORTUS_BACKGROUND=true
- PORTUS_BACKGROUND_REGISTRY_ENABLED=true
- PORTUS_BACKGROUND_SYNC_ENABLED=true
- PORTUS_BACKGROUND_SYNC_STRATEGY=update-delete
links:
- db
# env_file:
# - ./portus.env
volumes:
- traefik_certs_raw:/certificates:ro
networks:
- portus
db:
image: library/mariadb:10.0.33
command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0
# env_file:
# - ./portus.env
environment:
- MYSQL_DATABASE=portus_production
- MYSQL_ROOT_PASSWORD=${DATABASE_PASSWORD}
volumes:
- mariadb:/var/lib/mysql
networks:
- portus
# clair: TODO:
# image: quay.io/coreos/clair
# restart: unless-stopped
# depends_on:
# - postgres
# links:
# - postgres
# - portus
# ports:
# - "6060-6061:6060-6061"
# volumes:
# - /tmp:/tmp
# - ./clair/clair.yml:/clair.yml
# command: [-config, /clair.yml]
registry:
image: library/registry:2.7.1
# env_file:
# - ./portus.env
environment:
# REGISTRY_HTTP_ADDR: registry.ashlab.dev
# Authentication
REGISTRY_AUTH_TOKEN_REALM: https://portus.ashlab.dev/v2/token
REGISTRY_AUTH_TOKEN_SERVICE: registry.ashlab.dev
REGISTRY_AUTH_TOKEN_ISSUER: portus.ashlab.dev
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certificates/portus.ashlab.dev/certificate.crt
# Portus endpoint
REGISTRY_NOTIFICATIONS_ENDPOINTS: >
- name: portus
url: https://portus.ashlab.dev/v2/webhooks/events
timeout: 2000ms
threshold: 5
backoff: 1s
volumes:
- traefik_certs_raw:/certificates:ro
- registry:/var/lib/registry
- secrets:/secrets:ro
- ./config.yml:/etc/docker/registry/config.yml:ro
ports:
# - 5000:5000
- 5001:5001 # required to access debug service
links:
- portus:portus
networks:
- portus
- public
labels:
- "traefik.enable=true"
# - "traefik.http.middlewares.sslHeaders.headers.SSLHost=registry.ashlab.dev"
- "traefik.http.routers.registry.rule=Host(`registry.ashlab.dev`)"
- "traefik.http.routers.registry.middlewares=https_redirect, sslHeaders"
- "traefik.http.routers.registry.service=registry"
- "traefik.http.routers.registry.tls=true"
- "traefik.http.routers.registry.tls.certresolver=le"
- "traefik.http.services.registry.loadbalancer.server.port=5000"
- "traefik.http.services.registry.loadbalancer.server.scheme=http"
# - "traefik.http.middlewares.https_redirect.redirectscheme.scheme=https" # Standard move to default when traefik fixes behavior
# - "traefik.http.middlewares.https_redirect.redirectscheme.permanent=true"
# - "traefik.http.middlewares.sslHeaders.headers.framedeny=true"
# - "traefik.http.middlewares.sslHeaders.headers.sslredirect=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSSeconds=315360000"
# - "traefik.http.middlewares.sslHeaders.headers.browserXSSFilter=true"
# - "traefik.http.middlewares.sslHeaders.headers.contentTypeNosniff=true"
# - "traefik.http.middlewares.sslHeaders.headers.forceSTSHeader=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSIncludeSubdomains=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSPreload=true"
deploy:
labels:
- "traefik.enable=true"
# - "traefik.http.middlewares.sslHeaders.headers.SSLHost=registry.ashlab.dev"
- "traefik.http.routers.registry.rule=Host(`registry.ashlab.dev`)"
- "traefik.http.routers.registry.middlewares=https_redirect, sslHeaders"
- "traefik.http.routers.registry.service=registry"
- "traefik.http.routers.registry.tls=true"
- "traefik.http.routers.registry.tls.certresolver=le"
- "traefik.http.services.registry.loadbalancer.server.port=5000"
- "traefik.http.services.registry.loadbalancer.server.scheme=http"
# - "traefik.http.middlewares.https_redirect.redirectscheme.scheme=https" # Standard move to default when traefik fixes behavior
# - "traefik.http.middlewares.https_redirect.redirectscheme.permanent=true"
# - "traefik.http.middlewares.sslHeaders.headers.framedeny=true"
# - "traefik.http.middlewares.sslHeaders.headers.sslredirect=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSSeconds=315360000"
# - "traefik.http.middlewares.sslHeaders.headers.browserXSSFilter=true"
# - "traefik.http.middlewares.sslHeaders.headers.contentTypeNosniff=true"
# - "traefik.http.middlewares.sslHeaders.headers.forceSTSHeader=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSIncludeSubdomains=true"
# - "traefik.http.middlewares.sslHeaders.headers.STSPreload=true"
volumes:
secrets:
driver: local
driver_opts:
type: "none"
o: "bind,rw"
device: "/mnt/workspace/portus/secrets"
traefik_certs_raw:
driver: local
driver_opts:
type: "none"
o: "bind,ro"
device: "/mnt/workspace/traefik_certs_raw/"
mariadb:
registry:
networks:
public:
external: true
portus:
|
|
@robgiovanardi so apply environment variables I gave you, inserting them into your |
|
@Jean-Baptiste-Lasselle Hey - is there a reason why the latest garbage collection code (d847071) isn't in the 2.4.3 release? As far as I can tell the release was made in May, but this code was merged in January? This is a pretty crucial patch. Can we get a 2.4.4 release? |
hi Matt @diranged , Actually I am not an OpenSUSE engineer,or am not 'yet?) part of the official portus support or dev team. So I can't make release. Looks like you're gonna have to make a personal release n your infrastructure, building |
|
Hi @Jean-Baptiste-Lasselle thanks for your help. About environment variables, you probably read the doc: http://port.us.org/docs/Configuring-Portus.html:
So using is equal to use: Anyway I tested the env var but nothing changed: keep_latest was ignored and all tags was deleted. @diranged You're right, that fundamental commit isn't in the 2.4.3 release.. I'll try to build the latest release and coming back for a feedback. |
@robgiovanardi @diranged so thank you both of you I think you are right, the If so :
@robgiovanardi so thank you about :
So ok, we can infer which env. variables to use, from the config files descriptions :
My analysis to really understand where we are now (starting from
|
|
hi @robgiovanardi Did you try and use I mean :
|
|
Hi @Jean-Baptiste-Lasselle I still have no time to do my tests, but thanks for points me out that docker release. This will speed up my tests |
Very interesting test automation case though : Test Suite Bundle 1: short-term GC
Suite 1
export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE
### NODE ON ALPINE
docker pull node:8-alpine
docker tag node:8-alpine $OCI_SERVICE/hive/node:8-alpine
# docker logged-in
docker push $OCI_SERVICE/hive/node:8-alpine
### HELM ON ALPINE
docker pull alpine/helm:3.1.1
docker tag alpine/helm:3.1.1-alpine $OCI_SERVICE/hive/helm:3.1.1-alpine
docker push $OCI_SERVICE/hive/helm:3.1.1-alpine
### ATLANTIS TERRAGRUNT
docker pull exositebot/atlantis-terragrunt:version-1.4.1
docker tag exositebot/atlantis-terragrunt:version-1.4.1 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.1
docker push $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.1
export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE
# docker logged-in
### NODE ON ALPINE
docker pull node:9-alpine
docker tag node:9-alpine $OCI_SERVICE/hive/node:9-alpine
docker push $OCI_SERVICE/hive/node:9-alpine
### HELM ON ALPINE
docker pull alpine/helm:3.1.0
docker tag alpine/helm:3.1.0-alpine $OCI_SERVICE/hive/helm:3.1.0-alpine
docker push $OCI_SERVICE/hive/helm:3.1.0-alpine
### ATLANTIS TERRAGRUNT
docker pull exositebot/atlantis-terragrunt:version-1.4.0
docker tag exositebot/atlantis-terragrunt:version-1.4.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.0
docker tag $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.0
export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE
# docker logged-in
### NODE ON ALPINE
docker pull node:10-alpine
docker tag node:10-alpine $OCI_SERVICE/hive/node:10-alpine
docker push $OCI_SERVICE/hive/node:10-alpine
### HELM ON ALPINE
docker pull alpine/helm:3.0.3
docker tag alpine/helm:3.0.3-alpine $OCI_SERVICE/hive/helm:3.0.3-alpine
docker push $OCI_SERVICE/hive/helm:3.0.3-alpine
### ATLANTIS TERRAGRUNT
docker pull exositebot/atlantis-terragrunt:version-1.3.0
docker tag exositebot/atlantis-terragrunt:version-1.3.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.3.0
docker tag $OCI_SERVICE/hive/atlantis-terragrunt:version-1.3.0
export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE
# docker logged-in
### NODE ON ALPINE
docker pull node:11-alpine
docker tag node:11-alpine $OCI_SERVICE/hive/node:11-alpine
docker push $OCI_SERVICE/hive/node:11-alpine
### HELM ON ALPINE
docker pull alpine/helm:2.15.2
docker tag alpine/helm:2.15.2-alpine $OCI_SERVICE/hive/helm:2.15.2-alpine
docker push $OCI_SERVICE/hive/helm:2.15.2-alpine
### ATLANTIS TERRAGRUNT
docker pull exositebot/atlantis-terragrunt:version-1.1.0
docker tag exositebot/atlantis-terragrunt:version-1.1.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.1.0
docker tag $OCI_SERVICE/hive/atlantis-terragrunt:version-1.1.0
export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE
# docker logged-in
### NODE ON ALPINE
docker pull node:12-alpine
docker tag node:12-alpine $OCI_SERVICE/hive/node:12-alpine
docker push $OCI_SERVICE/hive/node:12-alpine
### HELM ON ALPINE
docker pull alpine/helm:2.15.1
docker tag alpine/helm:2.15.1-alpine $OCI_SERVICE/hive/helm:2.15.1-alpine
docker push $OCI_SERVICE/hive/helm:2.15.1-alpine
### ATLANTIS TERRAGRUNT
docker pull exositebot/atlantis-terragrunt:version-1.2.0
docker tag exositebot/atlantis-terragrunt:version-1.2.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.2.0
docker tag $OCI_SERVICE/hive/atlantis-terragrunt:version-1.2.0
export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE
### NODE ON ALPINE
docker pull node:8-alpine
docker pull node:9-alpine
docker pull node:10-alpine
docker pull node:11-alpine
docker pull node:12-alpine
docker tag node:8-alpine $OCI_SERVICE/hive/node:8-alpine
docker tag node:9-alpine $OCI_SERVICE/hive/node:9-alpine
docker tag node:10-alpine $OCI_SERVICE/hive/node:10-alpine
docker tag node:11-alpine $OCI_SERVICE/hive/node:11-alpine
docker tag node:12-alpine $OCI_SERVICE/hive/node:12-alpine
### HELM ON ALPINE
docker pull alpine/helm:3.1.1
docker pull alpine/helm:3.1.0
docker pull alpine/helm:3.0.3
docker pull alpine/helm:2.15.2
docker pull alpine/helm:2.15.1
docker tag alpine/helm:3.1.1-alpine $OCI_SERVICE/hive/helm:3.1.1-alpine
docker tag alpine/helm:3.1.0-alpine $OCI_SERVICE/hive/helm:3.1.0-alpine
docker tag alpine/helm:3.0.3-alpine $OCI_SERVICE/hive/helm:3.0.3-alpine
docker tag alpine/helm:2.15.2-alpine $OCI_SERVICE/hive/helm:2.15.2-alpine
docker tag alpine/helm:2.15.1-alpine $OCI_SERVICE/hive/helm:2.15.1-alpine
### ATLANTIS TERRAGRUNT
docker pull exositebot/atlantis-terragrunt:version-1.4.1
docker pull exositebot/atlantis-terragrunt:version-1.4.0
docker pull exositebot/atlantis-terragrunt:version-1.3.0
docker pull exositebot/atlantis-terragrunt:version-1.1.0
docker pull exositebot/atlantis-terragrunt:version-1.2.0
docker tag exositebot/atlantis-terragrunt:version-1.4.1 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.1
docker tag exositebot/atlantis-terragrunt:version-1.4.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.0
docker tag exositebot/atlantis-terragrunt:version-1.3.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.3.0
docker tag exositebot/atlantis-terragrunt:version-1.1.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.1.0
docker tag exositebot/atlantis-terragrunt:version-1.2.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.2.0
|
Shared Helper for load testsSo I have files with huge list of existing image tags : # preparation of huge test dataset for
# tests like loead tests on portus
export namespace=library
export repo_name=ubuntu
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=notary
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=centos
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=debian
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=archlinux
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=registry
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=node
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=busybox
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=dkron
export repo_name=dkron
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=httpd
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags | awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}'>> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=tomcat
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=golang
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=python
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=rails
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
export namespace=library
export repo_name=ruby
echo '{' > all.${namespace}.${repo_name}.tags.json
curl -L -s https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags| awk -F '[' '{print $2}'| awk -F ']' '{print $1}'| awk -F '},' '{for (i=0;i<=NF;i++) { if ($i == $NF) {print $i;} else {print $i "}, "} }}' >> all.${namespace}.${repo_name}.tags.json
echo '}' >> all.${namespace}.${repo_name}.tags.json
ls -allh *.tags.json
# And we can have 1,2,5,10,20, 30, 50, 100, 150, 200, 300, etc...500 simultaneous docker clients constantly pulling and pushing any randomly picked tag among this list |
|
Hi @Jean-Baptiste-Lasselle I tried to use portus:2.5 but got:
So I can't proceed with that. Anyway thanks for your support |
Hi @robgiovanardi It's just a pleasure to support users who feedback so fast and share. Plus I love the General issue with the Portus project, it's like very significant, I believe, about the revolution currently happening all over the world with the cloud. So, Ok, I duly note your results about So next: I'll reproduce those two issues #2197 and #2200 I am so not surprised by #2197 , because while solving this issue I found out that there is (February 2020) no rails official image based on ruby above One of the cloud's critical challenge : mastering the dependency hell. Note there's here a distribution management problem of the The dream problem for a devops like me. :) |
|
@robgiovanardi just to say i now read both :
All in all I'd say 99% chances I provide you with a fix for your setup, before end of next week |
|
Just to write it down,like that, I have a feeling openSUSE is conducted a huge migration on containers, so that it works with |
|
Also to share with community, the tools list I'm gonna test to manage batch jobs :
|
Jean-Baptiste-Lasselle
pushed a commit
to pokusio/opensuzie-oci-library
that referenced
this issue
Mar 1, 2020
Purpose of this Release : Sharing a finally successfully built `portus:2.5` container, workingon SUSE/Portus#2241
Jean-Baptiste-Lasselle
pushed a commit
to pokusio/opensuzie-oci-library
that referenced
this issue
Mar 1, 2020
Purpose of this Release : Sharing a finally successfully built `portus:2.5` container, workingon SUSE/Portus#2241
Jean-Baptiste-Lasselle
pushed a commit
to pokusio/opensuzie-oci-library
that referenced
this issue
Mar 1, 2020
Purpose of this Release : Sharing a finally successfully built `portus:2.5` container, workingon SUSE/Portus#2241
|
Hi @robgiovanardi I have news here :
background_1 | /usr/bin/bundle:23:in `load': cannot load such file -- /usr/lib64/ruby/gems/2.6.0/gems/bundler-1.16.4/exe/bundle (LoadError)
background_1 | from /usr/bin/bundle:23:in `<main>'
But there is that very strange thing I noticedAnd that, I note to bear it in mind :
portuscontainer | + export RACK_ENV=production
portuscontainer | + RACK_ENV=production
portuscontainer | + export RAILS_ENV=production
portuscontainer | + RAILS_ENV=production
portuscontainer | + export CCONFIG_PREFIX=PORTUS
portuscontainer | + CCONFIG_PREFIX=PORTUS
portuscontainer | + '[' -z '' ']'
portuscontainer | + export GEM_PATH=/srv/Portus/vendor/bundle/ruby/2.5.3
portuscontainer | + GEM_PATH=/srv/Portus/vendor/bundle/ruby/2.5.3
if [ -z "$PORTUS_GEM_GLOBAL" ]; then
export GEM_PATH="/srv/Portus/vendor/bundle/ruby/2.6.0"
fi
if [ -z "$PORTUS_GEM_GLOBAL" ]; then
export GEM_PATH="/srv/Portus/vendor/bundle/ruby/2.5.3"
fi
I guess I have my tomorrow 's TODO List. Funny thing Check out the commit (and its commit message) of @mssola on https://github.com/opensuse/docker-containers , which is today (1st of March 2020) :
:)
The logs from my
|
The Real Build from source : Additional info related to
|
Pipeline cycle identifiedOk, the OpenSUSE Team use the https://github.com/SUSE/Portus source code repo, to version control pipeline recipe, in particular :
|
Undocumented use of RVM confirmedLine 4 in 3de71a6 Line 5 in 3de71a6 Thought : I think they use RVM to normalize the set of executables involved in the portus ruby stack : They use it to make their |
Jean-Baptiste-Lasselle
pushed a commit
to pokusio/opensuzie-oci-library
that referenced
this issue
Mar 2, 2020
Fixed SUSE/Portus#2241 Hacked into OpenSUSE CI/CD The fix is not acceptable for a long run, the OpenSUSE packages must be updated to include a solution involving rubygems-integration new feaiture, to run specifically on OpenSUSE Leap, similar to the DEBIAN specific management describe here: rubygems/rubygems#2180 (comment) There could be something like a [SUSE_LEAP_DISABLE_RUBYGEMS_INTEGRATION] env.var.
Jean-Baptiste-Lasselle
pushed a commit
to pokusio/opensuzie-oci-library
that referenced
this issue
Mar 2, 2020
Fixed SUSE/Portus#2241 Hacked into OpenSUSE CI/CD The fix is not acceptable for a long run, the OpenSUSE packages must be updated to include a solution involving rubygems-integration new feaiture, to run specifically on OpenSUSE Leap, similar to the DEBIAN specific management describe here: rubygems/rubygems#2180 (comment) There could be something like a [SUSE_LEAP_DISABLE_RUBYGEMS_INTEGRATION] env.var.
Jean-Baptiste-Lasselle
pushed a commit
to pokusio/opensuzie-oci-library
that referenced
this issue
Mar 2, 2020
Fixed SUSE/Portus#2241 Hacked into OpenSUSE CI/CD The fix is not acceptable for a long run, the OpenSUSE packages must be updated to include a solution involving rubygems-integration new feaiture, to run specifically on OpenSUSE Leap, similar to the DEBIAN specific management describe here: rubygems/rubygems#2180 (comment) There could be something like a [SUSE_LEAP_DISABLE_RUBYGEMS_INTEGRATION] env.var. see also #1 (comment)
Victory :D@robgiovanardi Tested and re-tested, versioned and released, https://github.com/pokusio/opensuzie-oci-library/releases/tag/0.0.2 IMPORTANT UPDATE TO READER WILLING TO USE PORTUS 2.5 :
(end of update)
|
|
Now, I can't wait you test that on your side :) |
This was referenced Mar 2, 2020
Open
Closed
This was referenced Apr 11, 2020
|
Hi, I'm a bit confused. I'm setting up an instance of Portus. I've been battling with it for quite a while, the example compose file gets you so far but there has been a lot of trial and error to get things working. |
Hi @benthurley82 ,
Alright, so you HAVE, to use the
Now :
|
|
Thanks for all your contributions! |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Description
I Activated Garbage Collector on Portus Background process with
keep_latest: 5andolder_than: 100But it deletes all images older_than 100 ignoring the keep_latest flag. In result I have old repositories wiped all completely
Steps to reproduce
Here initial logs:
Deployment information
Deployment method: Portus is deployed as a standalone Container (not Compose) which connects to local MariaDB and Registry.
Configuration:
Portus version: 2.4.3@5a616c0ef860567df5700708256f42505cdb9952
env_portus: environment file used for customizing Portus Foreground:
We are running portus with:
env_background: environment file used for customizing Portus Background:
Then we are running portus background:
Thanks in advance
Roberto
The text was updated successfully, but these errors were encountered: