Got no project package links index.html using allowlist plugin

[JJLAAA]

my bandersnatch.conf:

[mirror]
directory = /opt/bandersnatch
json = false
release-files = true
cleanup = false
master = https://pypi.org
timeout = 1000
global-timeout = 3600
workers = 3
hash-index = false
stop-on-error = false
storage-backend = filesystem
; log-config = /opt/bandersnatch/bandersnatch-log.conf
; root_uri = https://example.com
verifiers = 3
; keep_index_versions = 0
; vim: set ft=cfg:
; diff-file = /srv/pypi/mirrored-files
; diff-append-epoch = true

[plugins]
enabled =
    blocklist_project
    blocklist_release
    allowlist_project
    allowlist_release
[blocklist]

; https://bandersnatch.readthedocs.io/en/latest/filtering_configuration.html
; https://pypi.org/stats/
[allowlist]
packages =
    cntk
    tensorflow-gpu
    tensorflow
    tensorflow-cpu
    torch

[cooperlees]

Hi,

Going to need a little more information here and logs. What:

• version of bandersnatch
• Version of python
• OS

Can you upload your logs somewhere please? Using grep to find “ERROR” lines is always a good start too.

Looks like you have a lot of plugins enabled you’re not using. There are interop bugs for sure as we don’t test multiple plugins enabled.

I’d suggest removing all plugins from being enabled except allowlist_project and re-run a full sync (bandersnatch mirror takes a —force-sync or like option)

If that does not work use “bandersnatch sync” for now on each project to unblock.

[cooperlees]

Here is the CI config that we make use of “allowlist_project” in (so it’s one of the more tested plugins): https://github.com/pypa/bandersnatch/blob/master/src/bandersnatch/tests/ci.conf

[JJLAAA]

Hi,

Going to need a little more information here and logs. What:

• version of bandersnatch
• Version of python
• OS

Can you upload your logs somewhere please? Using grep to find “ERROR” lines is always a good start too.

Looks like you have a lot of plugins enabled you’re not using. There are interop bugs for sure as we don’t test multiple plugins enabled.

I’d suggest removing all plugins from being enabled except allowlist_project and re-run a full sync (bandersnatch mirror takes a —force-sync or like option)

If that does not work use “bandersnatch sync” for now on each project to unblock.

version of bandersnatch: docker image: supercmmetry/bandersnatch id: 1bf84c4e0d3a
OS: ubuntu 20.04

the only error is "ERROR: Unable to load entry point swift_plugin = bandersnatch_storage_plugins.swift:SwiftStorage: No module named 'keystoneauth1'"

[JJLAAA]

Here is the CI config that we make use of “allowlist_project” in (so it’s one of the more tested plugins): https://github.com/pypa/bandersnatch/blob/master/src/bandersnatch/tests/ci.conf

I tried this conf,and I got json file but there still no .whl file in simple dir

[cooperlees]

supercmmetry/bandersnatch is not a supported Docker image. I also can't find it (only https://github.com/supercmmetry/bandersnatch). You sure you're not using https://hub.docker.com/r/pypa/bandersnatch? If not can you try

• docker pull pypa/bandersnatch:4.4.0

This will get you on a known version and we can more easily debug from there. If that does not work, please try:

• docker pull pypa/bandersnatch:latest
  and we can debug the code in master and try fix your bug.

Without any errors in your logs, this will be hard to debug. Can you run with --debug too please and see if that helps show any issue.

Here is our CI job that is passing, using allowlist_project and has checks to ensure certain version of black + some other packages JSON etc. exist.
https://github.com/pypa/bandersnatch/blob/master/test_runner.py

I tried this conf,and I got json file but there still no .whl file in simple dir

The simple dir will only ever have html in it. The packages dir should get sdists + wheel (.whl) files.

[JJLAAA]

supercmmetry/bandersnatch is not a supported Docker image. I also can't find it (only https://github.com/supercmmetry/bandersnatch). You sure you're not using https://hub.docker.com/r/pypa/bandersnatch? If not can you try

• docker pull pypa/bandersnatch:4.4.0

This will get you on a known version and we can more easily debug from there. If that does not work, please try:

• docker pull pypa/bandersnatch:latest
  and we can debug the code in master and try fix your bug.

Without any errors in your logs, this will be hard to debug. Can you run with --debug too please and see if that helps show any issue.

Here is our CI job that is passing, using allowlist_project and has checks to ensure certain version of black + some other packages JSON etc. exist.
https://github.com/pypa/bandersnatch/blob/master/test_runner.py

I tried this conf,and I got json file but there still no .whl file in simple dir

The simple dir will only ever have html in it. The packages dir should get sdists + wheel (.whl) files.

thanks for your reply!
pypa/bandersnatch:latest id: e1b11990ea3e
is there a way the simple dir show the link for .whl file? like this

[cooperlees]

Sorry, I thought you mean in the directory. Yes, of course the HTML should have links to the sdists (.zip + tgz) and .whl files.

[JJLAAA]

Here is the CI config that we make use of “allowlist_project” in (so it’s one of the more tested plugins): https://github.com/pypa/bandersnatch/blob/master/src/bandersnatch/tests/ci.conf

I got these links in simple page, but why the letters (c、t) is the first-level directory? How can i set these (cntk、tensorflow-gpu..) link as the first-level directory? thank you!
here is my conf :

[mirror]
directory = /opt/bandersnatch
json = true
cleanup = true
master = https://pypi.org
timeout = 60
global-timeout = 18000
workers = 3
hash-index = true
stop-on-error = true
storage-backend = filesystem
verifiers = 3
keep_index_versions = 2

[plugins]
enabled =
    allowlist_project
[allowlist]
packages =
    cntk
    tensorflow-gpu
    tensorflow
    tensorflow-cpu
    torch

[cooperlees]

That’s just a config setting you have set.
(More information here: https://bandersnatch.readthedocs.io/en/latest/mirror_configuration.html#hash-index)

Change the following line:

• hash-index = false

Then run:

• “rm -r simple/*”
• Re-run “bandersnatch mirror —force”
  (Or what ever the full run argument is so all your packages update and bandersnatch does not try an incremental update)

So what was the fix? Is there a bug we need to fix or was it just configuration error?

[JJLAAA]

I think it's just a network problem. After wait for hours, it finally worked. Thanks for your support,I will close these issue