Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[V5] Verdaccio is not working behind http2 proxy #2190

Closed
viceice opened this issue Apr 23, 2021 · 27 comments · Fixed by #2271
Closed

[V5] Verdaccio is not working behind http2 proxy #2190

viceice opened this issue Apr 23, 2021 · 27 comments · Fixed by #2271

Comments

@viceice
Copy link

viceice commented Apr 23, 2021

Describe the bug

Verdaccio v5 is not detecting headers passed from reverse proxy, I think it's because of http2 is lowercasing all headers

log

{
  "level": 25,
  "time": 1619156769309,
  "pid": 7,
  "hostname": "verdaccio-5bdb8cd95f-6d78j",
  "req":
  {
    "method": "GET",
    "url": "/",
    "query": {},
    "params": {},
    "headers":
    {
      "host": "npm.domain.com",
      "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36",
      "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "accept-encoding": "gzip, deflate, br",
      "accept-language": "en-US,en;q=0.9,de;q=0.8",
      "cache-control": "no-cache",
      "pragma": "no-cache",
      "sec-ch-ua": "\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"",
      "sec-ch-ua-mobile": "?0",
      "sec-fetch-dest": "document",
      "sec-fetch-mode": "navigate",
      "sec-fetch-site": "none",
      "sec-fetch-user": "?1",
      "upgrade-insecure-requests": "1",
      "x-forwarded-for": "172.30.2.20",
      "x-forwarded-host": "npm.domain.com",
      "x-forwarded-port": "443",
      "x-forwarded-proto": "https",
      "x-forwarded-server": "internal-traefik-6d4789455-hnklc",
      "x-real-ip": "172.30.2.20"
    },
    "remoteAddress": "10.42.235.33",
    "remotePort": 38724
  },
  "ip": "10.42.235.33",
  "msg": "@{ip} requested '@{req.method} @{req.url}'"
}

Woraround: override VERDACCIO_PUBLIC_URL

To Reproduce

running verdaccio behind traefik reverse proxy, which is terminating https (http2) connection.

Expected behavior

Verdaccio should detect lowercase reverse proxy headers.

Screenshots

Configuration File (cat ~/.config/verdaccio/config.yaml)

Environment information

Debugging output

  • $ NODE_DEBUG=request verdaccio display request calls (verdaccio <--> uplinks)
  • $ DEBUG=express:* verdaccio enable extreme verdaccio debug mode (verdaccio api)
  • $ npm -ddd prints:
  • $ npm config get registry prints:
@viceice
Copy link
Author

viceice commented Apr 23, 2021

https://tools.ietf.org/html/rfc7540

8.1.2.  HTTP Header Fields

   HTTP header fields carry information as a series of key-value pairs.
   For a listing of registered HTTP headers, see the "Message Header
   Field" registry maintained at <https://www.iana.org/assignments/
   message-headers>.

   Just as in HTTP/1.x, header field names are strings of ASCII
   characters that are compared in a case-insensitive fashion.  However,
   header field names MUST be converted to lowercase prior to their
   encoding in HTTP/2.  A request or response containing uppercase
   header field names MUST be treated as malformed (Section 8.1.2.6).

@juanpicado
Copy link
Member

Thanks @viceice , good catch.
Issue is here
https://github.com/verdaccio/verdaccio/blob/5.x/src/lib/constants.ts#L32
https://github.com/verdaccio/verdaccio/blob/5.x/src/lib/utils.ts#L644

Also could be use https://github.com/verdaccio/verdaccio/blob/5.x/docs/env.variables.md#verdaccio_forwarded_proto
env variable VERDACCIO_FORWARDED_PROTO

is the unique header affected? or there are others?

@juanpicado juanpicado added this to the Bugs milestone Apr 23, 2021
@viceice
Copy link
Author

viceice commented Apr 23, 2021

Proto was wrong too. It return http ip address urls instead of https and domain. But I saw it detects the original remote ip header.

Did not tested additional env vars.

@juanpicado
Copy link
Member

Ok thanks for the feedback, I will figure out a solution.

@viceice
Copy link
Author

viceice commented Apr 23, 2021

Thanks, no hurry, as there is a workaround 🙃

@FStefanni
Copy link

Hi,

I am having the same issue, and the workaround seems to not work for me.
My scenario is also with Traefik v2 and it is as follows:

  • Traefik provides https, on port 4873
  • In the docker compose, I have added: VERDACCIO_PUBLIC_URL=https://my.server.com:4873 as environment variable
  • Verdaccio listen on 4873, in http

But the result is that it goes in Gateway Timeout.
Could someone help please? What am I missing to make the workaround work properly?

Thank you.
Regards.

@viceice
Copy link
Author

viceice commented Apr 26, 2021

Please provide you compose file. I think you have an error there. Maybe traefik is trying https to verdaccio.

@FStefanni
Copy link

Hi,

sure. Traefik:

version: "3.7"
services:
  shared-traefik:
    image: traefik:v2.3.6
    container_name: shared-traefik
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "2"
    ports:
      - "80:80"
      - "443:443"
      - "5000:5000"
      - "4873:4873"
    command:
      - --log.level=INFO
      - --api=true
      - --api.insecure=true
      - --api.dashboard=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entryPoint.to=websecure
      - --entrypoints.web.http.redirections.entryPoint.scheme=https
      - --entrypoints.web.http.redirections.entryPoint.priority=1
      - --entrypoints.web.http.redirections.entryPoint.permanent=true
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.http.tls=true
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --entrypoints.registry.address=:5000
      - --entrypoints.registry.http.tls=true
      - --entrypoints.registry.http.tls.certresolver=myresolver
      - --entrypoints.verdaccio.address=:4873
      - --entrypoints.verdaccio.http.tls=true
      - --entrypoints.verdaccio.http.tls.certresolver=myresolver
      - --certificatesresolvers.myresolver.acme.httpchallenge=true
      - --certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web
      #- --certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
      - --certificatesresolvers.myresolver.acme.email=my@email.com
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
      - --providers.file.directory=/traefik-config
      - --providers.file.watch=true
      - --serverstransport.insecureskipverify=true
    networks:
      .aaa_shared_network:
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /usr/share/docker/shared/traefik/acme-v2.json:/letsencrypt/acme.json
      - /usr/share/docker/shared/traefik/config:/traefik-config
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.dashboard.loadbalancer.server.port=8080"
      - "traefik.http.services.dashboard.loadbalancer.server.scheme=http"
      - "traefik.http.routers.dashboard.rule=Host(`my.server.com`) && (PathPrefix(`/shared/traefik`) || Headers(`Referer`, `https://my.server.com/shared/traefik/dashboard/`))"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.tls.certresolver=myresolver"
      - "traefik.http.routers.dashboard.middlewares=dashboard@docker"
      - "traefik.http.middlewares.dashboard.chain.middlewares=dashboard-auth,dashboard-strip"
      - "traefik.http.middlewares.dashboard-auth.basicauth.users=admin:password"
      - "traefik.http.middlewares.dashboard-strip.stripprefix.prefixes=/shared/traefik"
      - "traefik.http.middlewares.dashboard-strip.stripprefix.forceslash=true"

Verdaccio:

  shared-verdaccio:
    image: verdaccio/verdaccio:5.0.1
    container_name: shared-verdaccio
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "2"
    environment:
        - VERDACCIO_PORT=4873
        - VERDACCIO_PUBLIC_URL=https://my.server.com:4873
    #ports:
    #    - "4873:4873"
    volumes:
        #- /etc/timezone:/etc/timezone:ro
        #- /etc/localtime:/etc/localtime:ro
        - /usr/share/docker/shared/verdaccio/storage:/verdaccio/storage
        - /usr/share/docker/shared/verdaccio/conf:/verdaccio/conf
        - /usr/share/docker/shared/verdaccio/plugins:/verdaccio/plugins
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.shared-verdaccio.loadbalancer.server.port=4873"
      - "traefik.http.services.shared-verdaccio.loadbalancer.server.scheme=http"
      - "traefik.http.routers.shared-verdaccio.rule=Host(`my.server.com`) || Headers(`Referer`, `https://my.server.com:4873`)"
      - "traefik.http.routers.shared-verdaccio.entrypoints=verdaccio"
      - "traefik.http.routers.shared-verdaccio.tls=true"
      - "traefik.http.routers.shared-verdaccio.tls.certresolver=myresolver"

Regards.

@viceice
Copy link
Author

viceice commented Apr 26, 2021

Looks like an issue with your traefik config. You should extend logging and ask on traefik issue tracker for help.

@FStefanni
Copy link

FStefanni commented Apr 27, 2021

Hi

So do you confirm that this is how you set the VERDACCIO_PUBLIC_URL, and it worked for you?
If so, I suppose I have to wait for Verdaccio to fix this issue.

Regards

@viceice
Copy link
Author

viceice commented Apr 27, 2021

@FStefanni Yes, but i do not use a custom port. I use normal 443 and let treafik rules select right backend. I also use kubernetes instead of docker, but it shouldn't change anything about env variables.

@qballer
Copy link

qballer commented May 5, 2021

This seems to be an interesting issue to take a crack at, can someone help with traefik configuration to see if I can fix this as I'm unfamiliar with this proxy. Surely it will help the next person if I don't get a PR going.

@FStefanni
Copy link

FStefanni commented May 6, 2021

Hi,

an example config is what I posted up.
It serves verdaccio on port 4873, with TLS powered by let's encrypt.
This means it requires to have a valid public domain and ip.
Traefik dashboard is provided on /shared/traefik, port 443.

I can help you to tune traefik config for your tests.

For example:

  1. Remove the let's encrypt cert, and use a self signed: just comment the following lines:

       - --certificatesresolvers.myresolver.acme.httpchallenge=true
       - --certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web
       #- --certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
       - --certificatesresolvers.myresolver.acme.email=my@email.com
       - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
    
  2. Change the occurences of my.server.com to localhost

  3. Set a valid password for traefik dashboard, or disable authentication. To remove auth:

    Replace:
      - "traefik.http.middlewares.dashboard.chain.middlewares=dashboard-auth,dashboard-strip"
      - "traefik.http.middlewares.dashboard-auth.basicauth.users=admin:password"
      - "traefik.http.middlewares.dashboard-strip.stripprefix.prefixes=/shared/traefik"
      - "traefik.http.middlewares.dashboard-strip.stripprefix.forceslash=true"
    
    With:
       - "traefik.http.middlewares.dashboard.stripprefix.prefixes=/shared/traefik"
       - "traefik.http.middlewares.dashboard.stripprefix.forceslash=true"
    

If you need other, let us know.

Regards.

@viceice
Copy link
Author

viceice commented May 6, 2021

@FStefanni better to provide a simple docker-compose.yml with minimal config. For testing the default traefik generated cert would be enough. Traefik dashboard is not needed for testing, but a local host alias for a custom domain will help

@FStefanni
Copy link

Hi,

sorry but I have no time to test this.
So, by just "cuttng" and adapting the prev configs, I post the following.
No sure it is 100% fine:

version: "3.7"
services:
  shared-traefik:
    image: traefik:v2.3.6
    container_name: shared-traefik
    ports:
      - "80:80"
      - "443:443"
    command:
      - --log.level=INFO
      - --api=true
      - --api.insecure=true
      - --api.dashboard=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entryPoint.to=websecure
      - --entrypoints.web.http.redirections.entryPoint.scheme=https
      - --entrypoints.web.http.redirections.entryPoint.priority=1
      - --entrypoints.web.http.redirections.entryPoint.permanent=true
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.http.tls=true
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --serverstransport.insecureskipverify=true
    networks:
      .aaa_shared_network:
     
  shared-verdaccio:
    image: verdaccio/verdaccio:5.0.1
    container_name: shared-verdaccio
    environment:
        - VERDACCIO_PORT=4873
        - VERDACCIO_PUBLIC_URL=https://my.server.com:4873
    volumes:
        - /usr/share/docker/shared/verdaccio/storage:/verdaccio/storage
        - /usr/share/docker/shared/verdaccio/conf:/verdaccio/conf
        - /usr/share/docker/shared/verdaccio/plugins:/verdaccio/plugins
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.shared-verdaccio.loadbalancer.server.port=4873"
      - "traefik.http.services.shared-verdaccio.loadbalancer.server.scheme=http"
      - "traefik.http.routers.shared-verdaccio.rule=Host(`my.server.com`)"
      - "traefik.http.routers.shared-verdaccio.tls=true"
      - "traefik.http.routers.shared-verdaccio.tls.certresolver=myresolver"

All http routes are redirected to https.
Now vedaccio should work via http/https (80/443).
So this should work since it contains the "workaround" by @viceice, but I have not tested it.
Still need to adapt (or disable) verdaccio volumes mount points, and the url of the server (my.server.com).

Regards

@viceice
Copy link
Author

viceice commented May 7, 2021

Here a more basic sample, but it didn't reproduce the issue 😕

# /etc/hosts: 127.0.0.1 npm.example.com
services:
  traefik:
    image: traefik:v2.3.6
    ports:
      - "443:443"
    command:
      - --log.level=INFO
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.http.tls=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  verdaccio:
    image: verdaccio/verdaccio:5.0.4
    environment:
    - NODE_ENV=production
    #        - VERDACCIO_PUBLIC_URL=https://npm.example.com
    volumes:
      - verdaccio-data:/verdaccio
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.verdaccio.loadbalancer.server.port=4873"
      - "traefik.http.routers.verdaccio.rule=Host(`npm.example.com`)"
      - "traefik.http.routers.verdaccio.tls=true"

volumes:
  verdaccio-data:

log

{
  "level": 25,
  "time": 1620370390553,
  "pid": 9,
  "hostname": "b17920ecb8eb",
  "req":
  {
    "method": "GET",
    "url": "/",
    "query": {},
    "params": {},
    "headers":
    {
      "host": "npm.example.com",
      "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36",
      "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "accept-encoding": "gzip, deflate, br",
      "accept-language": "en-US,en;q=0.9,de-DE;q=0.8,de;q=0.7",
      "cache-control": "max-age=0",
      "dnt": "1",
      "sec-ch-ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"",
      "sec-ch-ua-mobile": "?0",
      "sec-fetch-dest": "document",
      "sec-fetch-mode": "navigate",
      "sec-fetch-site": "cross-site",
      "sec-fetch-user": "?1",
      "upgrade-insecure-requests": "1",
      "x-forwarded-for": "172.18.0.1",
      "x-forwarded-host": "npm.example.com",
      "x-forwarded-port": "443",
      "x-forwarded-proto": "https",
      "x-forwarded-server": "b0b08bafbfcd",
      "x-real-ip": "172.18.0.1"
    },
    "remoteAddress": "172.18.0.2",
    "remotePort": 34344
  },
  "ip": "172.18.0.2",
  "msg": "@{ip} requested '@{req.method} @{req.url}'"
}

Using docker desktop on windows with wsl v2. Also tested traefik v2.4.8.

@juanpicado juanpicado linked a pull request May 22, 2021 that will close this issue
@juanpicado
Copy link
Member

@viceice
You can give it a try, if you are using docker.

docker run -it --rm --name verdaccio -p 4873:4873 verdaccio/verdaccio:jota-fix-2190

@viceice
Copy link
Author

viceice commented May 22, 2021

Sure, but earliest next week. Sorry.

@juanpicado juanpicado unpinned this issue May 23, 2021
juanpicado added a commit that referenced this issue May 26, 2021
* fix: improve request header handling

* chore: fix test

* chore: apply suggestion
@zkochan
Copy link
Collaborator

zkochan commented Jun 2, 2021

Seems like a fix was merged to v6 #2271

Can we expect a fix to be released in v5 as well?

@juanpicado
Copy link
Member

Mmm should be on v5.1.0. Where is not yet available on v6 alpha yet.

@juanpicado
Copy link
Member

I will close this one, since I consider this fix if anyone has issues feel free to open a new ticket and will check it out again.

Verdaccio 5 automation moved this from To do to Done Jun 5, 2021
@FStefanni
Copy link

Hi,

just to confirm that verdaccio:5.1.0 works fine behind Traefik.
Thank you for the effort.

Regards.

@juanpicado
Copy link
Member

@zkochan was right, the commit was not included on 5.1.0 rather on https://github.com/verdaccio/verdaccio/releases/tag/v5.1.1 which was just published

@viceice
Copy link
Author

viceice commented Jul 9, 2021

Sorry, this is not resolved for me with 5.1.1. 😕
image

logs

[
  {
    "level": 25,
    "time": 1625840623871,
    "pid": 8,
    "hostname": "verdaccio-754d946f9b-bzbqt",
    "req":
    {
      "method": "GET",
      "url": "/",
      "query": {},
      "params": {},
      "headers":
      {
        "host": "npm.domain.test",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
        "accept-encoding": "gzip, deflate, br",
        "accept-language": "en-US,en;q=0.9,de;q=0.8",
        "sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"",
        "sec-ch-ua-mobile": "?0",
        "sec-fetch-dest": "document",
        "sec-fetch-mode": "navigate",
        "sec-fetch-site": "none",
        "sec-fetch-user": "?1",
        "upgrade-insecure-requests": "1",
        "x-forwarded-for": "172.30.2.20",
        "x-forwarded-host": "npm.domain.test",
        "x-forwarded-port": "443",
        "x-forwarded-proto": "https",
        "x-forwarded-server": "internal-traefik-549b78ffcf-8gc62",
        "x-real-ip": "172.30.2.20"
      },
      "remoteAddress": "10.42.103.58",
      "remotePort": 45910
    },
    "ip": "10.42.103.58",
    "msg": "@{ip} requested '@{req.method} @{req.url}'"
  }
  {
    "level": 25,
    "time": 1625840623873,
    "pid": 8,
    "hostname": "verdaccio-754d946f9b-bzbqt",
    "request":
    {
      "method": "GET",
      "url": "/"
    },
    "user": null,
    "remoteIP": "172.30.2.20 via 10.42.103.58",
    "status": 200,
    "bytes":
    {
      "in": 0,
      "out": 492
    },
    "msg": "@{status}, user: @{user}(@{remoteIP}), req: '@{request.method} @{request.url}', bytes: @{bytes.in}/@{bytes.out}"
  }
]

@uguumur64
Copy link

Sorry, this is not resolved for me with 5.1.1. 😕 image

logs

[
  {
    "level": 25,
    "time": 1625840623871,
    "pid": 8,
    "hostname": "verdaccio-754d946f9b-bzbqt",
    "req":
    {
      "method": "GET",
      "url": "/",
      "query": {},
      "params": {},
      "headers":
      {
        "host": "npm.domain.test",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
        "accept-encoding": "gzip, deflate, br",
        "accept-language": "en-US,en;q=0.9,de;q=0.8",
        "sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"",
        "sec-ch-ua-mobile": "?0",
        "sec-fetch-dest": "document",
        "sec-fetch-mode": "navigate",
        "sec-fetch-site": "none",
        "sec-fetch-user": "?1",
        "upgrade-insecure-requests": "1",
        "x-forwarded-for": "172.30.2.20",
        "x-forwarded-host": "npm.domain.test",
        "x-forwarded-port": "443",
        "x-forwarded-proto": "https",
        "x-forwarded-server": "internal-traefik-549b78ffcf-8gc62",
        "x-real-ip": "172.30.2.20"
      },
      "remoteAddress": "10.42.103.58",
      "remotePort": 45910
    },
    "ip": "10.42.103.58",
    "msg": "@{ip} requested '@{req.method} @{req.url}'"
  }
  {
    "level": 25,
    "time": 1625840623873,
    "pid": 8,
    "hostname": "verdaccio-754d946f9b-bzbqt",
    "request":
    {
      "method": "GET",
      "url": "/"
    },
    "user": null,
    "remoteIP": "172.30.2.20 via 10.42.103.58",
    "status": 200,
    "bytes":
    {
      "in": 0,
      "out": 492
    },
    "msg": "@{status}, user: @{user}(@{remoteIP}), req: '@{request.method} @{request.url}', bytes: @{bytes.in}/@{bytes.out}"
  }
]

@viceice have you found a solution? I am in same situation

@viceice
Copy link
Author

viceice commented May 23, 2024

still using the VERDACCIO_PUBLIC_URL workaround. never tested again.

@uguumur64
Copy link

@viceice Thank you so much, VERDACCIO_PUBLIC_URL worked for me!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Verdaccio 5
  
Done
Development

Successfully merging a pull request may close this issue.

6 participants