failed to fetch layers: unexpected status code: 403 Forbidden #1370

pflin-tw · 2021-09-17T08:08:46Z

pflin-tw
Sep 17, 2021

Hello,

I run the clair with COMBO mode, and the service is running normally.
Now, I used clairctl to scan image, but it failed to fetch layers with 403 status code.
I've no idea about it, the following is what I tried.

I tried scan my local registry image:
clairctl -D -c clair_config/config.yaml report --host http://172.18.19.120:8181 localhost:5000/pgadmin4:5.7
However it failed with:

2021-09-17T10:28:19+08:00 DBG using text output
2021-09-17T10:28:19+08:00 DBG fetching ref=localhost:5000/pgadmin4:5.7
2021-09-17T10:28:24+08:00 DBG found manifest digest=sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a ref=localhost:5000/pgadmin4:5.7
2021-09-17T10:28:24+08:00 DBG  method=GET path=/indexer/api/v1/index_report/sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a status="403 Forbidden"
2021-09-17T10:28:24+08:00 DBG index error error="unexpected return status: 403" ref=localhost:5000/pgadmin4:5.7
2021-09-17T10:28:24+08:00 ERR  error="unexpected return status: 403"

Then, I tried to get manifest:
clairctl -D -c clair_config/config.yaml manifest localhost:5000/pgadmin4:5.7
It works, and I got the response:

2021-09-17T15:26:16+08:00 DBG manifest
2021-09-17T15:26:16+08:00 DBG fetching name=localhost:5000/pgadmin4:5.7
2021-09-17T15:26:16+08:00 DBG found manifest digest=sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a ref=localhost:5000/pgadmin4:5.7
2021-09-17T15:26:16+08:00 DBG found layers count=12 ref=localhost:5000/pgadmin4:5.7
2021-09-17T15:26:16+08:00 DBG ok name=localhost:5000/pgadmin4:5.7
{"hash":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","layers":[{"hash":"sha256:e396e8b032cf2e91792dafe9f2aa2b1c94b015bbb1938525b1c59f661f73a8a4","uri":"http://localhost:5000/v2/pgadmin4/blobs/sha256:e396e8b032cf2e91792dafe9f2aa2b1c94b015bbb1938525b1c59f661f73a8a4","headers":{}},
....
{"hash":"sha256:3fd9c52a5161a5e329213cbc36bc4b3e1fa53e0f4e04c322b216ba9100c61714","uri":"http://localhost:5000/v2/pgadmin4/blobs/sha256:3fd9c52a5161a5e329213cbc36bc4b3e1fa53e0f4e04c322b216ba9100c61714","headers":{}}]}

I used curl & Postman to test the api to index the above manifest:
curl -X POST -d '{"hash":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","layers":[{"hash":"sha256:e396e8b032cf2e91792dafe9f2aa2b1c94b015bbb1938525b1c59f661f73a8a4","uri":"http://localhost:5000/v2/pgadmin4/blobs/sha256:e396e8b032cf2e91792dafe9f2aa2b1c94b015bbb1938525b1c59f661f73a8a4","headers":{}}, .... {"hash":"sha256:3fd9c52a5161a5e329213cbc36bc4b3e1fa53e0f4e04c322b216ba9100c61714","uri":"http://localhost:5000/v2/pgadmin4/blobs/sha256:3fd9c52a5161a5e329213cbc36bc4b3e1fa53e0f4e04c322b216ba9100c61714","headers":{}}]}' http://172.18.19.120:8181/indexer/api/v1/index_report -H 'Accept: application/json' -H 'Content-Type: application/json'
I got the Postman response:

{
    "manifest_hash": "sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a",
    "state": "IndexError",
    "packages": {},
    "distributions": {},
    "repository": {},
    "environments": {},
    "success": false,
    "err": "failed to fetch layers: encountered error while fetching a layer: fetcher: unexpected status code: 403 Forbidden (body starts: \"<HTML><HEAD>\\r\\n<TITLE>Access Denied</TITLE>\\r\\n</HEAD>\\r\\n<BODY>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n<big><strong></strong></big><BR>\\r\\n</FONT>\\r\\n\\r\\n<TABLE border=0 cellPadding=1 width=\\\"80%\\\">\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n<big>Access Denied (policy_denied)\")"
}

Server side error log:

{"level":"info","component":"libindex/Libindex.Index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","time":"2021-09-17T09:47:14Z","message":"index request start"}
{"level":"debug","component":"rhel/repo2cpe/UpdatingMapper.do","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","url":"https://www.redhat.com/security/data/metrics/repository-to-cpe.json","version":"1.1","time":"2021-09-17T09:47:14Z","message":"attempting fetch of repo2cpe mapping file"}
{"level":"debug","component":"rhel/repo2cpe/UpdatingMapper.do","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","url":"https://www.redhat.com/security/data/metrics/repository-to-cpe.json","version":"1.1","time":"2021-09-17T09:47:14Z","message":"atomic update of local mapping file complete"}
{"level":"debug","component":"libindex/Libindex.index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","time":"2021-09-17T09:47:14Z","message":"locking"}
{"level":"debug","component":"libindex/Libindex.index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","time":"2021-09-17T09:47:14Z","message":"locked"}
{"level":"info","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","state":"CheckManifest","time":"2021-09-17T09:47:14Z","message":"starting scan"}
{"level":"info","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","state":"CheckManifest","time":"2021-09-17T09:47:14Z","message":"manifest to be scanned"}
{"level":"info","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","state":"FetchLayers","time":"2021-09-17T09:47:14Z","message":"layers fetch start"}
{"level":"debug","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","state":"FetchLayers","count":12,"time":"2021-09-17T09:47:14Z","message":"fetching layers"}
{"level":"debug","component":"internal/indexer/fetcher/fetcher.fetch","layer":"sha256:3fd9c52a5161a5e329213cbc36bc4b3e1fa53e0f4e04c322b216ba9100c61714","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","state":"FetchLayers","uri":"http://172.18.19.120:5000/v2/pgadmin4/blobs/sha256:3fd9c52a5161a5e329213cbc36bc4b3e1fa53e0f4e04c322b216ba9100c61714","time":"2021-09-17T09:47:14Z","message":"layer fetch start"}
....
{"level":"debug","component":"internal/indexer/fetcher/fetcher.fetch","layer":"sha256:3b3a46def8beb3512a91648b18a3bd00fa23c96eb5afd3e2df66205f257eadc5","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","state":"FetchLayers","uri":"http://172.18.19.120:5000/v2/pgadmin4/blobs/sha256:3b3a46def8beb3512a91648b18a3bd00fa23c96eb5afd3e2df66205f257eadc5","time":"2021-09-17T09:47:14Z","message":"layer fetch start"}
{"level":"debug","component":"internal/indexer/fetcher/fetcher.fetch","layer":"sha256:41d99922e290dd59c3997fb1931865150d5b8570324314d3c3f40d36853a0c5a","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","state":"FetchLayers","uri":"http://172.18.19.120:5000/v2/pgadmin4/blobs/sha256:41d99922e290dd59c3997fb1931865150d5b8570324314d3c3f40d36853a0c5a","time":"2021-09-17T09:47:14Z","message":"layer fetch start"}
{"level":"warn","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","state":"FetchLayers","error":"encountered error while fetching a layer: fetcher: unexpected status code: 403 Forbidden (body starts: \"<HTML><HEAD>\\r\\n<TITLE>Access Denied</TITLE>\\r\\n</HEAD>\\r\\n<BODY>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n<big><strong></strong></big><BR>\\r\\n</FONT>\\r\\n<blockquote>\\r\\n<TABLE border=0 cellPadding=1 width=\\\"80%\\\">\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n<big>Access Denied (policy_denied)\")","time":"2021-09-17T09:47:14Z","message":"layers fetch failure"}
{"level":"info","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","state":"FetchLayers","time":"2021-09-17T09:47:14Z","message":"layers fetch done"}
{"level":"error","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","state":"CheckManifest","error":"failed to fetch layers: encountered error while fetching a layer: fetcher: unexpected status code: 403 Forbidden (body starts: \"<HTML><HEAD>\\r\\n<TITLE>Access Denied</TITLE>\\r\\n</HEAD>\\r\\n<BODY>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n<big><strong></strong></big><BR>\\r\\n</FONT>\\r\\n<blockquote>\\r\\n<TABLE border=0 cellPadding=1 width=\\\"80%\\\">\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n<big>Access Denied (policy_denied)\")","time":"2021-09-17T09:47:14Z","message":"error during scan"}
{"level":"debug","component":"libindex/Libindex.index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","time":"2021-09-17T09:47:14Z","message":"unlocked"}
{"level":"info","component":"libindex/Libindex.Index","manifest":"sha256:f1dfa0691ed9e0af9049e9ba20327fff1a132215aacb697462e3888967e6f65a","time":"2021-09-17T09:47:14Z","message":"index request done"}
{"level":"info","component":"httptransport/New","remote_addr":"172.18.19.113:30209","method":"POST","request_uri":"/indexer/api/v1/index_report","status":201,"time":"2021-09-17T09:47:14Z","message":"handled HTTP request"}
{"level":"debug","component":"notifier/Delivery.deliver","deliverer":"webhook","id":2,"time":"2021-09-17T09:47:39Z","message":"delivery tick"}

I don't know if there is setting I missed?

The config.yaml

introspection_addr: :8989
http_listen_addr: :8181
log_level: debug
indexer:
  connstring: host=172.18.19.120 port=5432 user=clair password=clairpass dbname=clair sslmode=disable
  scanlock_retry: 10
  layer_scan_concurrency: 5
  migrations: true
matcher:
user=clair password=clairpass dbname=clair sslmode=disable
  max_conn_pool: 100
  run: ""
  migrations: true
  updater_sets:
  - "alpine"
  - "aws"
  - "debian"
  - "oracle"
  - "photon"
  - "pyupio"
  - "rhel"
  - "suse"
  - "ubuntu"
matchers:
  names:
  - "alpine"
  - "aws"
  - "debian"
  - "oracle"
  - "photon"
  - "python"
  - "rhel"
  - "suse"
  - "ubuntu"
  - "crda"
  config:
    crda:
      url: https://f8a-analytics-preview-2445582058137.production.gw.apicast.io/?user_key=3e42fa66f65124e6b1266a23431e3d08
notifier:
  indexer_addr: http://clair-indexer:8181/
matcher_addr: http://clair-matcher:8181/
  connstring: host=172.18.19.120 port=5432 user=clair password=clairpass dbname=clair sslmode=disable
  migrations: true
  delivery_interval: 1m
  poll_interval: 5m
  webhook:
    target: "http://webhook/"
    callback: "http://clair-notifier/notifier/api/v1/notifications"
  amqp:
    exchange:
        name: ""
        type: "direct"
        durable: true
        auto_delete: false
    uris: ["amqp://user:pass@host:10000/vhost"]
    direct: false
    routing_key: "notifications"
    callback: "http://clair-notifier/notifier/api/v1/notifications"
    tls:
     root_ca: "optional/path/to/rootca"
     cert: "madatory/path/to/cert"
     key: "madatory/path/to/key"
  stomp:
    desitnation: "notifications"
    direct: false
    callback: "http://clair-notifier/notifier/api/v1/notifications"
    login:
      login: "username"
      passcode: "passcode"
    tls:
     root_ca: "optional/path/to/rootca"
     cert: "madatory/path/to/cert"
     key: "madatory/path/to/key"

trace:
  name: "jaeger"
  probability: 1
  jaeger:
    agent:
      endpoint: "localhost:6831"
    service_name: "clair"

metrics:
  name: "prometheus"

Thanks in advence.

Answered by hdonnay

Sep 21, 2021

Is "localhost" the same for both the client and the clair instance? You'll notice that the manifest has a URL of where to fetch the layer from -- make sure that's resolvable from the clair container.

View full answer

hdonnay · 2021-09-21T21:40:55Z

hdonnay
Sep 21, 2021
Maintainer

Is "localhost" the same for both the client and the clair instance? You'll notice that the manifest has a URL of where to fetch the layer from -- make sure that's resolvable from the clair container.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

failed to fetch layers: unexpected status code: 403 Forbidden #1370

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

Select a reply

failed to fetch layers: unexpected status code: 403 Forbidden #1370

pflin-tw Sep 17, 2021

Replies: 1 comment

hdonnay Sep 21, 2021 Maintainer

pflin-tw
Sep 17, 2021

hdonnay
Sep 21, 2021
Maintainer