Docker upgrade to 4.0.0 not working, missing all packages

[johan-smits]

Describe the bug
Docker upgrade to 4.0.0 not working, missing all packages.
The UI did not show any packages and also install a package did not work

To Reproduce
Steps to reproduce the behavior:

1. Go to Update the docker image to verdaccio/verdaccio:4.0.0
2. Open the website
3. No packages available

Expected behavior
See all NPM packages

Kubernetes (please complete the following information):

• Docker verdaccio tag: verdaccio/verdaccio:4.0.0

Configuration File (cat ~/.config/verdaccio/config.yaml)
$ cat /verdaccio/conf/config.yaml

storage: /verdaccio/storage/
max_body_size: 1000mb
auth:
  htpasswd:
    file: ./htpasswd
uplinks:
  npmjs:
    url: https://registry.npmjs.org/
packages:
  '@*/*':
    access: $all
    publish: $authenticated
    proxy:
    storage: '/verdaccio/storage'
  '**':
    access: $all
    publish: $authenticated
    proxy:
    storage: '/verdaccio/storage'
logs:
  - {type: stdout, format: pretty, level: debug}
#  - {type: stdout, format: pretty, level: http}


security:
   token:
      web: 24h # by default
      api: never # by defaukt
   algorithm: HS256 # by default

Additional context
Spoke on Discord and got these links: https://verdaccio.org/blog/2019/02/24/migrating-verdaccio and https://verdaccio.org/blog/2019/05/13/the-new-docker-image-verdaccio-4

None where clear to me what the changes are, I use kubernetes with no ENV variables and separate volume mounts. What are the real breaking changes?

---

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: npm-registry
  namespace: backoffice
  name: npm-registry
spec:
  # Stop old container before starting new one
  strategy:
    type: Recreate
    rollingUpdate: null
  selector:
    matchLabels:
      app: npm-registry
  replicas: 1
  template:
    metadata:
      labels:
        app: npm-registry
    spec:
      containers:
      - name: npm-registry
        image: verdaccio/verdaccio:4.0.0
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 300m
            memory: 200Mi
          limits:
            cpu: 300m
            memory: 250Mi
        ports:
        - containerPort: 4873
          name: http
        volumeMounts:
        - mountPath: /verdaccio/storage
          name: npmdata-rook-ceph-block
          subPath: storage
        - mountPath: /verdaccio/plugins
          name: npmdata-rook-ceph-block
          subPath: plugins
        - mountPath: /verdaccio/conf
          name: npmdata-rook-ceph-block
          subPath: conf
        readinessProbe:
          httpGet:
            path: /
            port: 4873
          initialDelaySeconds: 5
          periodSeconds: 20
        livenessProbe:
          httpGet:
            path: /
            port: 4873
          timeoutSeconds: 2
      volumes:
      - name: npmdata-rook-ceph-block
        persistentVolumeClaim:
          claimName: npmdata-rook-ceph-block

[sbusch]

TL;DR: workaround below

I confirm this behavior, tested with 4.0.1

I bind-mount a host directory into the container at /verdaccio, which worked fine on 3.x.

On a 4.0.1 container, the contents of /verdaccio are then:

• /verdaccio/conf and /verdaccio/plugins contain the contents from host, but
• /verdaccio/storage contains only an auto-generated .verdaccio-db.json.

I think its because the Dockerfile for 3.x and 4.x define volumes differently:

• 3.x: VOLUME /verdaccio: https://github.com/verdaccio/verdaccio/blob/3.x/Dockerfile#L46
• 4.x: VOLUME /verdaccio/storage: https://github.com/verdaccio/verdaccio/blob/master/Dockerfile#L54)

I'm no Docker expert so no idea if our issue is caused by a Docker bug, quirk or if it's defined behavior.

Workaround (fixed the problem for me): define three bind mounts for /verdaccio/conf, /verdaccio/plugins and /verdaccio/storage separately.

Disadvantages: (verbose and) error-prone in case of new directories in the future

---

Update: found the reasoning for the change in Dockerfile: #836 (comment) and #836 (comment)

@dlouzan in #836 you wrote "I'm not sure about this involving breaking changes". I'm OK with the change but it's breaking and should be documented at https://verdaccio.org/blog/2019/02/24/migrating-verdaccio#migrating-from-verdaccio-3x-to-verdaccio-4x

[johan-smits]

I'm using Kubernetes and your workaround did not work for me. As it was my default already.

[johnruck]

Work around did not work for me as well (I was also already doing 3 independent volume mounts. I have also just tried new mount directories with no effect. One thing I have noticed is that I think it might be permissions on the mount directories that is the problem (for point of reference, I let Docker create the new directories and then copied in the config.yaml file after it created them). Attaching the docker instance gets me the following output on the new directories on my first attempt to use the "new" server.

 http --> 200, req: 'GET https://registry.npmjs.org/@angular%2Fcli' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/@types%2Fjasmine' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/@angular%2Fcli', bytes: 0/1414339
 error--- unexpected error: EACCES: permission denied, mkdir '/verdaccio/storage/@angular'
Error: EACCES: permission denied, mkdir '/verdaccio/storage/@angular'
 http <-- 500, user: null(172.16.121.185), req: 'GET /@angular%2fcli', error: internal server error
 http --> 200, req: 'GET https://registry.npmjs.org/@types%2Fjasmine', bytes: 0/133927
 error--- unexpected error: EACCES: permission denied, mkdir '/verdaccio/storage/@types'
Error: EACCES: permission denied, mkdir '/verdaccio/storage/@types'

note this is with the "Latest" docker image. Verdaccio web ui says version 4.01.

Any suggestions? I have even tried turning off the caching and it still outputs these errors and fails.

In case it matters - this was on Ubuntu 18.04.2 LTS

command used:

docker run -d --rm -it -v /home/john/verdaccio4/conf:/verdaccio/conf -v /home/john/verdaccio4/storage:/verdaccio/storage -v /home/john/verdaccio4/plugins:/verdaccio/plugins -p 4873:4873 --name npm-server verdaccio/verdaccio

image info
verdaccio/verdaccio                latest              d13802d13b71        2 weeks ago         116MB

[sbusch]

@johnruck are you using docker-compose (like me) or Kubernetes (as @johan-smits)?

I'm running Docker Desktop on current macOS

[johnruck]

I am just using the docker structure they provided - docker pull verdaccio/verdaccio:latest

I try to be as simple as I can :)

[johnruck]

hmm, in looking through the migration document, they say

VERDACCIO_USER_UID | 10001 | the user id being used to apply folder permissions

does this mean that I need to have a user with id of 10001 on the host machine with permissions on the Docker mount directories? (I havent really completely figured out docker and if/how it maps permissions between the docker image instance and the host on volumes yet so I am wondering if this is why verdaccio cant create the directories).

Right now the new directories are root:root ownership on the host filesystem. I just checked host /etc/passwd, there is no user with id 10001.

UPDATE- well, I tried passing in via the -e for VERDACCIO_USER_NAME and VERDACCIO_USER_ID to match my user credential information and I am still getting the same permission denied on the mkdir commands so that doesnt seem to be the cure.

[johnruck]

Ok, this time I did a docker run without the -d, here is all of the initial output up until the first errors

latest: Pulling from verdaccio/verdaccio
Digest: sha256:f607a698d200fefab6613a4ab609aac88223712fcaa4b5c1301d74d70a73d64e
Status: Image is up to date for verdaccio/verdaccio:latest
 warn --- config file  - /verdaccio/conf/config.yaml
 warn --- Plugin successfully loaded: verdaccio-htpasswd
 warn --- Plugin successfully loaded: verdaccio-audit
 warn --- http address - http://0.0.0.0:4873/ - verdaccio/4.0.1
 http --> 200, req: 'GET https://registry.npmjs.org/@angular%2Flanguage-service' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/@angular%2Fcompiler-cli' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/@types%2Fjasminewd2' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/jasmine-spec-reporter' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/@types%2Fjasminewd2', bytes: 0/12107
 http --> 200, req: 'GET https://registry.npmjs.org/@angular%2Fcli' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/@types%2Fnode' (streaming)
 error--- unexpected error: EACCES: permission denied, mkdir '/verdaccio/storage/@types'
Error: EACCES: permission denied, mkdir '/verdaccio/storage/@types'
 http <-- 500, user: null(172.16.121.185), req: 'GET /@types%2fjasminewd2', error: internal server error
 http --> 200, req: 'GET https://registry.npmjs.org/@types%2Fjasmine' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/jasmine-core' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/codelyzer' (streaming)
 http --> 200, req: 'GET https://registry.npmjs.org/jasmine-core', bytes: 0/56418
 error--- unexpected error: EACCES: permission denied, mkdir '/verdaccio/storage/jasmine-core'
Error: EACCES: permission denied, mkdir '/verdaccio/storage/jasmine-core'
 http <-- 500, user: null(172.16.121.185), req: 'GET /jasmine-core', error: internal server error

[johnruck]

In case it helps, my config.yaml (originated with verdaccio 3):

#
# This is the config file used for the docker images.
# It allows all users to do anything, so don't use it on production systems.
#
# Do not configure host and port under `listen` in this file
# as it will be ignored when using docker.
# see https://verdaccio.org/docs/en/docker#docker-and-custom-port-configuration
#
# Look here for more config file examples:
# https://github.com/verdaccio/verdaccio/tree/master/conf
#

# path to a directory with all packages
storage: /verdaccio/storage
# path to a directory with plugins to include
plugins: /verdaccio/plugins

web:
  # WebUI is enabled as default, if you want disable it, just uncomment this line
  #enable: false
  title: Verdaccio

auth:
  htpasswd:
    file: /verdaccio/conf/htpasswd
    # Maximum amount of users allowed to register, defaults to "+infinity".
    # You can set this to -1 to disable registration.
    #max_users: 1000

# a list of other known repositories we can talk to
uplinks:
  npmjs:
    url: https://registry.npmjs.org/

packages:
  '@*/*':
    # scoped packages
    access: $all
    publish: $authenticated
    proxy: npmjs

  '**':
    # allow all users (including non-authenticated users) to read and
    # publish all packages
    #
    # you can specify usernames/groupnames (depending on your auth plugin)
    # and three keywords: "$all", "$anonymous", "$authenticated"
    access: $all

    # allow all known users to publish packages
    # (anyone can register by default, remember?)
    publish: $authenticated

    # if package is not available locally, proxy requests to 'npmjs' registry
    proxy: npmjs

# To use `npm audit` uncomment the following section
middlewares:
  audit:
    enabled: true

# log settings
logs:
  - {type: stdout, format: pretty, level: http}
  #- {type: file, path: verdaccio.log, level: info}

[dlouzan]

I'm joining late as usual :-/

@sbusch Yeah you're right, I wasn't sure at the time and this seems to be causing some headaches to users. We'll need to document this migration better. Named volumes have been the preferred solution recommended by Docker itself for some time now, but this is not directly compatible with bind mounts that people were using.

@johnruck Your issue is most probably caused because of using a bind mount instead of a volume mount for the storage. I'm doing some local tests to paste here a small guide.

[dlouzan]

@johnruck To be clear: you shouldn't need to change any env variables, just copy the contents of your bind mount to a named volume and reference the volume for verdaccio. As I said, I'm making some tests locally.

[johnruck]

I am using bind mounts because they make sense to me (since there is easy access to the files in the host file system and they are in a place where I know where the files are). I still haven't figured out named volumes yet - any tutorials you can point me at that make sense? :)

[dlouzan]

@johnruck Ok please try this:

Please test creating a named volume and copying the contents of your storage directory in the host to it. Then start your verdaccio referencing this volume.

PLEASE backup your storage directory before trying this, just in case, it shouldn't break anything but you never know.

# Create a docker volume for storage
docker volume create verdaccio-storage

# Run a temp container to mount your local storage directory
# Copy the storage to the named volume
docker run -it --rm -v verdaccio-storage:/verdaccio/storage -v /home/john/verdaccio4/storage:/backup ubuntu bash -c "cp -a /backup/. /verdaccio/storage"

# Start your verdaccio referencing the volume
docker run -d --rm -it -v /home/john/verdaccio4/conf:/verdaccio/conf -v verdaccio-storage:/verdaccio/storage -v /home/john/verdaccio4/plugins:/verdaccio/plugins -p 4873:4873 --name npm-server verdaccio/verdaccio

[johnruck]

ok, before I switch over this a few questions,

Is the volume create a one time thing, redo on reboot thing?

Any side effects of switching to these volumes after doing docker stop commands (I have bash scripts that I use to start and stop docker instances, I am wondering if using named volumes causes any oddities in the behavior - I am paranoid since before I found the --rm arguments for run I was having to do the docker rm commands before doing another start (and the reason why I have start/stop bash scropts to make sure I did all of the necessary stop steps).

Background - so far I have been using Docker as an easier virtual machine system, I havent really gotten too deep into it yet.

And doing a docker volume list on my machine comes up with over 50 numbered volumes that I have no idea what they are from :) which is the other reason I havent done named volumes before.

[dlouzan]

For now you don't need to "switch over", the commands I added do not touch your existing storage directly, apart from making a copy into a named volume.

Named volumes are the preferred approach for mounting external, persistent directories into containers. They have several advantages, between them that they are portable between operating systems and allow locations other than local (e.g. NFS or AWS S3).

The main difference in day to day is that you don't mount a specific directory in your filesystem. You work with the docker volume command, you can inspect any of them and you'll see they're located in a specific directory in your OS (in my case in macOS under /var/lib/docker/volumes).

[johnruck]

I hate to say this, no difference in the behavior

john@tohsaka:~$ ./start-npm
latest: Pulling from verdaccio/verdaccio
Digest: sha256:f607a698d200fefab6613a4ab609aac88223712fcaa4b5c1301d74d70a73d64e
Status: Image is up to date for verdaccio/verdaccio:latest
warn --- config file - /verdaccio/conf/config.yaml
warn --- Plugin successfully loaded: verdaccio-htpasswd
warn --- Plugin successfully loaded: verdaccio-audit
warn --- http address - http://0.0.0.0:4873/ - verdaccio/4.0.1
http --> 304, req: 'GET https://registry.npmjs.org/jasmine-core' (streaming)
http --> 304, req: 'GET https://registry.npmjs.org/jasmine-core', bytes: 0/0
http --> 304, req: 'GET https://registry.npmjs.org/codelyzer' (streaming)
http --> 304, req: 'GET https://registry.npmjs.org/codelyzer', bytes: 0/0
http --> 304, req: 'GET https://registry.npmjs.org/@types%2Fjasminewd2' (streaming)
http --> 304, req: 'GET https://registry.npmjs.org/@types%2Fjasminewd2', bytes: 0/0
error--- unexpected error: EACCES: permission denied, open '/verdaccio/storage/@types/jasminewd2/package.json.tmp6425228679378139'
Error: EACCES: permission denied, open '/verdaccio/storage/@types/jasminewd2/package.json.tmp6425228679378139'
http <-- 500, user: null(172.16.121.185), req: 'GET /@types%2fjasminewd2', error: internal server error
http --> 304, req: 'GET https://registry.npmjs.org/jasmine-spec-reporter' (streaming)
http --> 304, req: 'GET https://registry.npmjs.org/jasmine-spec-reporter', bytes: 0/0
error--- unexpected error: EACCES: permission denied, open '/verdaccio/storage/jasmine-core/package.json.tmp5282991988866828'
Error: EACCES: permission denied, open '/verdaccio/storage/jasmine-core/package.json.tmp5282991988866828'

start-npm contents:

john@tohsaka:~$ cat start-npm
#!/bin/sh

docker pull verdaccio/verdaccio:latest

#docker run -d --rm -it -v /home/john/verdaccio/conf:/verdaccio/conf -v /home/john/verdaccio/storage:/verdaccio/storage -v /home/john/verdaccio/plugins:/verdaccio/plugins -p 4873:4873 --name npm-server verdaccio/verdaccio

docker run --rm -it -v /home/john/verdaccio4/conf:/verdaccio/conf -v verdaccio-storage:/verdaccio/storage -v /home/john/verdaccio4/plugins:/verdaccio/plugins -p 4873:4873 --name npm-server verdaccio/verdaccio

#docker run -d --rm -it -v /home/john/verdaccio/storage:/verdaccio/storage -v /home/john/verdaccio/plugins:/verdaccio/plugins -p 4873:4873 --name npm-server verdaccio/verdaccio


using your same command line with 
ubuntu bash -c "ls -al /verdaccio/storage"  
instead shows the files with weird ownerships:

drwxr-xr-x    2 _apt  101  4096 Jun 12 13:03 yargs
drwxr-xr-x    2 _apt  101  4096 Jun 12 13:03 yargs-parser
drwxr-xr-x    2 _apt  101  4096 Jun 12 13:02 yeast
drwxr-xr-x    2 _apt  101  4096 Jun 12 13:01 yn
drwxr-xr-x    2 _apt  101  4096 Jun 12 13:01 zone.js

NOTE - I used my older verdaccio 3 storage directory instead of the newer verdaccio 4 one that was still empty (mainly because I have some private packages in there that I am hoping not to lose).

[johnruck]

hmm, I just redid the volume creation without the copy attempt and that seems to be happy. So on an internals question, what do I need to do to get my old private packages back into this new instance ? Is it as easy as copy in the package subdirectories from my old storage location and chown the files to 10001:65533 ownership ?

for reference, the same ls -al with verdaccio readding the packages:

drwxr-xr-x 2 10001 65533 4096 Jun 12 18:02 yargs
drwxr-xr-x 2 10001 65533 4096 Jun 12 18:02 yargs-parser
drwxr-xr-x 2 10001 65533 4096 Jun 12 18:03 yeast
drwxr-xr-x 2 10001 65533 4096 Jun 12 18:02 yn
drwxr-xr-x 2 10001 65533 4096 Jun 12 18:02 zone.js

[dlouzan]

@johnruck Well I'm not sure why your storage has the id of the _apt user, I guess they are what your local bind mount permissions are? Probably in my instructions we'll have to add another command for setting the permissions.

But yes, it should be as easy as to copy & chown on the volume contents.

[johnruck]

Ok, I have gotten back to broken state that I was in before when I accidentally triggered the version 4 update, I guess I will pick up that debugging in the other issue's existing ticket.

Thank you for your guidance on getting past this issue at least.

Btw - copying in the files got them into verdaccio, but it doesnt know they are private packages at the moment (verdaccio says I have no packages published yet) - that will clear whenever I push the next package build though.

[amoscatelli]

I confirm this issue with 4.0.4

[juanpicado]

Does anyone with enough Docker knowledge in volumes could help with this migration problem? A short guide how to do it would help.

[johan-smits]

@juanpicado we are also still stuck at the older version. It is strange that it is an issue due to the volumes are just mount points.

[savokiss]

I've moved Verdaccio from 3.x to 4.3，and encountered the permission denied issue. In case someone need it, just use this to give storage to user verdaccio in the container.

sudo chown -R 10001:65533 storage

For more info: https://medium.com/@mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf

[ishowman]

I've moved Verdaccio from 3.x to 4.3，and encountered the permission denied issue. In case someone need it, just use this to give storage to user verdaccio in the container.

sudo chown -R 10001:65533 storage

For more info: https://medium.com/@mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf

I using docker exec -it <container ID> /bin/sh to enter the running container, and input the command you offered, get message "/bin/sh: sudo: not found".

[savokiss]

@ishowman This command should be executed on the host mathine.

[johan-smits]

@savokiss this did not work for me.

Here is some log output:

 warn --- config file  - /verdaccio/conf/config.yaml
 warn --- Verdaccio started
 debug--- [local-storage/_sync]: init sync database
 debug--- [local-storage/_sync]: folder /verdaccio/storage created succeed
 debug--- [local-storage/_sync/writeFileSync]: sync write succeed
 debug--- [local-storage/_sync]: init sync database
 debug--- [local-storage/_sync]: folder /verdaccio/storage created succeed
 debug--- [local-storage/_sync/writeFileSync]: sync write succeed
 warn --- Plugin successfully loaded: verdaccio-htpasswd
 debug--- [local-storage/readPackage] read a package: @lws/react-matrix-sdk
...

/verdaccio/storage/@lws/react-matrix-sdk $ ls -ln
total 464
-rw-r--r-- 1 10001 65533 5323 Oct 1 12:07 package.json
-rw-r--r-- 1 10001 65533 464440 Oct 1 12:07 react-matrix-sdk-0.1.0-dev8974.tgz

But the interface shows that no packages are published yet.

[johan-smits]

I have it solved 🎉

In my config I had:

packages:
  '@*/*':
    access: $all
    publish: $authenticated
    proxy:
    storage: '/verdaccio/storage'
  '**':
    access: $all
    publish: $authenticated
    proxy:
    storage: '/verdaccio/storage'

And now it works with:

packages:
  '@*/*':
    access: $all
    publish: $authenticated
  '**':
    access: $all
    publish: $authenticated

And by removing the proxy and storage lines it worked. Followed the example

Also the chown was needed to fix the permissions as @ishowman suggested. It was a double issue with file permissions and config.

My complete config is now:

storage: /verdaccio/storage
max_body_size: 1000mb
auth:
  htpasswd:
    file: ./htpasswd

uplinks:
  npmjs:
    url: https://registry.npmjs.org/

packages:
  '@*/*':
    access: $all
    publish: $authenticated
  '**':
    access: $all
    publish: $authenticated

logs:
  - {type: stdout, format: pretty, level: debug}

security:
  legacy: true
  token:
     web: 24h # by default
     api: never # by defaukt
  algorithm: HS256 # by default
  api:
    jwt:
      sign:
        expiresIn: 60d
        notBefore: 0
  web:
    sign:
      expiresIn: 7d

Also looked at issue #1481 for the authentication.

[lock]

🤖This thread has been automatically locked 🔒 since there has not been any recent activity after it was closed.
We lock tickets after 90 days with the idea to encourage you to open a ticket with new fresh data and to provide you better feedback 🤝and better visibility 👀.
If you consider, you can attach this ticket 📨 to the new one as a reference for better context.
Thanks for being a part of the Verdaccio community! 💘