Rancher's local system-library is empty with volume mount.

[bentastic27]

What kind of request is this (question/bug/enhancement/feature request):
bug

Steps to reproduce (least amount of steps as possible):

Simply spinning up Rancher with the following triggers it:

docker run -d --restart=unless-stopped -p 80:80 -p 443:443 -v /opt/rancher:/var/lib/rancher -e CATTLE_SYSTEM_CATALOG=bundled rancher/rancher:v2.3.0

You can see that it's empty with the following:

$ docker exec -ti <RANCHER CONTAINER> ls management-state/local-catalogs/system-library/

Result:

The system-library directory will be empty when you mount /var/lib/rancher and use the CATTLE_SYSTEM_CATALOG variable. Rancher will also complain about the directory being empty when managing system charts.

Other details that may be helpful:

This doesn't seem to happen when not mounting /var/lib/rancher.

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):
  2.3.0
• Installation option (single install/HA):
  single

[dnoland1]

This also seems to be an issue with a rancher upgrade which uses --volumes-from rancher-data.

[sangeethah]

With thsi fix, catalog cache is moved from /var/lib/rancher to /var/lib/rancher-data

Scenario to validate:

Scenario 1:
Create a single node rancher server install using v2.2.8 .
Create a 1 cluster and deploy apps.

Upgrade rancher server to v2.3.0 using -e CATTLE_SYSTEM_CATALOG=bundled

docker run -d --volumes-from rancher-data \
  --restart=unless-stopped \
    -p 80:80 -p 443:443 \
    rancher/rancher:<RANCHER_VERSION_TAG> -e CATTLE_SYSTEM_CATALOG=bundled

Once rancher server is upgraded , we will not be able to see any system catalogs. ( Reproducing this issue).

Upgrade rancher server to v2.3.1-rc5 using -e CATTLE_SYSTEM_CATALOG=bundled
Once rancher server is upgraded , we will not be able to see system catalogs.
Should be able to enable monitoring.

Scenario 2:
Create a single node rancher server install using v2.2.8 .
Create a 1 cluster and deploy apps.
Upgrade rancher server to v2.3.1-rc5 using -e CATTLE_SYSTEM_CATALOG=bundled
Once rancher server is upgraded , we will not be able to see all system catalogs.
Should be able to enable monitoring.

Scenario 3:
Fresh install of v2.3.0 using volume-mount option and -e CATTLE_SYSTEM_CATALOG=bundled to reproduce the issue.

Scenario 4:
Fresh install of v2.3.1-rc5 using volume-mount option and -e CATTLE_SYSTEM_CATALOG=bundled. We should be able to see the all system catalogs. Should be able to enable monitoring.

[bmdepesa]

rancher/rancher:v2.3.1-rc5

Scenario 1:

• v2.2.8 install
• Create cluster
• Deploy apps
• Upgrade to v2.3.0 using bundled catalog, volumes from v2.2.8
• Catalogs show that local library is empty
• Upgrade to v2.3.1-rc5 using bundled catalog
• Catalogs appear, monitoring can be enabled

Scenario 2:

• v2.2.8 install
• Create cluster
• Deploy apps
• Upgrade to v2.3.1-rc5 using bundled catalog, volumes from v2.2.8
• Catalogs appear, monitoring can be enabled

Scenario 3:
Not covered (reproduced elsewhere)

Scenario 4:

• v2.3.1-rc5 fresh install, volume mount for /var/lib/rancher, and bundled catalogs
• Cluster is created
• Catalogs are present
• Monitoring can be enabled

In all cases, the local charts are available in /var/lib/rancher-data/local-catalogs after upgrading to v2.3.1-rc5