[ERROR] Not possible to create apt repo mirror using JFrog artifactory or GCP Artifact Registry

[bouh2]

Describe your bug
Apt repo https://deb.nodesource.com/ is no longer supporting folder listing feature as folder listing is encrypted by gpg key.
As result, it's no longer possible to mirror apt repo https://deb.nodesource.com/ using JFrog artifactory or GCP Artifact Registry.
Is it possible to support folder listing without encryption ? The could resolve a lot of issues for mirroring.

Have seen the mirror guide - https://github.com/nodesource/distributions/wiki/Creating-a-Repository-Mirror:-A-Step%E2%80%90by%E2%80%90Step-Guide#setting-up-a-deb-mirror-
It's stick the mirroring to 'aptly' for Debian systems. But, there are many other mirroring tools which are used more widely and more popular.

Distribution Information:

• OS: Debian/Ubuntu
• Mirroring tools: JFrog artifactory, GCP Artifact Registry.

Node Version:

• Node: Node v18.x, Node v20.x, Node v21.x

[riosje]

Hi @bouh2 currently we can't support folder listing because we're using an Object Storage solution to store the packages.

there are many other mirroring tools which are used more widely and more popular.
What tool are you trying to implement?
What errors are you getting from jfrog or GCP artifact?

Can you share more details about your implementation?

[bouh2]

Hi @riosje
Thanks for looking after this issue !

Here are details:

1. JFrog Artifactory configuration. Remote repo is configured as https://deb.nodesource.com/node_20.x
   
   JFrog artifactory is able to test connection to the remote repo by downloading remote repo folder listing. Obviously, the test fails.
2. Apt client configuration on Ubuntu 20.04
   2a) gpg key has been downloaded and saved in /usr/share/keyrings/nodesource.gpg

$ ls -al /usr/share/keyrings/ | grep nodesource
-rw-r--r--   1 root root x x x nodesource.gpg

2b) apt list configuration

$ cat /etc/apt/sources.list.d/nodesource.list 
deb [arch=amd64 signed-by=/usr/share/keyrings/nodesource.gpg] https://artifactory/artifactory/xxxxxx-nodesource20 nodistro main

2c) checking sudo apt update:

$ sudo apt update
Get:1 https://artifactory/artifactory/ubuntu-remote focal-updates InRelease [114 kB]
Get:2 https://artifactory/artifactory/ubuntu-remote focal-backports InRelease [108 kB]
Get:3 https://artifactory/artifactory/ubuntu-remote  focal-security InRelease [114 kB]
Get:4 https://artifactory/artifactory/ubuntu-canonical focal InRelease [12.1 kB]
Err:5 https://artifactory/artifactory/xxxxxx-nodesource20 nodistro InRelease
  401  Unauthorized [IP: *.*.*.* 443]
Hit:6 https://artifactory/artifactory/microsoft focal InRelease
Reading package lists... Done
E: Failed to fetch https://artifactory/artifactory/xxxxxx-nodesource20/dists/nodistro/InRelease  401  Unauthorized [IP: *.*.*.* 443]
E: The repository 'https://artifactory/artifactory/xxxxxx-nodesource20 nodistro InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Apt updated packages from all remote repos except from xxxxxx-nodesource20 remote repo.

[riosje]

Hi @bouh2 I was able to create the remote repo using jFrog(SaaS version) and installed the packages sucesfully (nodejs and nsolid) without the folder listing.
I'll try to update the wiki with this setup later, in the meantime give you some screenshots with the config that I used.

This is the artifacts.

Basic config
Repo Name: Nodesource
Include Patterns: **
Repo Layout: simple-default
List Remote folder items: Uncheck

Advance Config

Repo config /etc/apt/sources.list

deb [trusted=yes signed-by=/usr/share/keyrings/nodesource.gpg] https://myuser:pwd@XXXXX.jfrog.io/artifactory/Nodesource nodistro main

[riosje]

@bouh2 also double checking your logs output, your issue is related to permissions in the jfrog side
The HTTP code 401 you got there represents an access denied.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

E: Failed to fetch https://artifactory/artifactory/xxxxxx-nodesource20/dists/nodistro/InRelease  401  Unauthorized [IP: *.*.*.* 443]

[riosje]

this is the updated wiki with the instruction to setup a mirror using jFrog artifactory.
https://github.com/nodesource/distributions/wiki/Creating-a-Repository-Mirror:-A-Step%E2%80%90by%E2%80%90Step-Guide#using-jfrog-artifactory

[bouh2]

@riosje It was an issue with old JFrog Artifactory. Permissions were set correctly. Once it was updated to one of the latest versions, issue is gone.
Thanks a lot for the detailed instructions and updated wiki ! Much appreciated ! 👍

[axqd]

@riosje could you please also update the steps for RPM mirror using jFrog Artifactory? Thanks.

[riosje]

@axqd it should work the same way.
give it a try and share the outcome....

[axqd]

@riosje Yes, I can confirm it works the same way. Guess still better to update the wiki. Without directory listing, it took me some effort to figure out the URL for RPM (i.e. https://rpm.nodesource.com/pub_20.x instead of node_20.x).