[BUG] SCCACHE_S3_NO_CREDENTIALS=true stopped working in 0.5.0

[ajschmidt8]

It seems that the SCCACHE_S3_NO_CREDENTIALS=true configuration stopped working in 0.5.0.

When using the following configuration with a publicly readable bucket with sccache version 0.4.2, sccache compiles correctly:

SCCACHE_BUCKET=<publicly_readable_bucket>
SCCACHE_IDLE_TIMEOUT=32768
SCCACHE_REGION=us-east-2
SCCACHE_S3_NO_CREDENTIALS=true
SCCACHE_S3_USE_SSL=true

However, in the same environment, sccache version 0.5.0 throws the following error message:

sccache: error: Server startup failed: cache storage failed to read: PermissionDenied (temporary) at read => no valid credential found, please check configuration or try again
    
    Context:
        service: s3
        path: .sccache_check
        range: 0-
    
    Run with SCCACHE_LOG=debug SCCACHE_NO_DAEMON=1 to get more information

The issue exists in 0.5.3 as well.

Setting SCCACHE_LOG=debug and SCCACHE_NO_DAEMON=1 did not seem to provide any additional details.

[ajschmidt8]

@Xuanwo, I'm wondering if the opendal upgrade could have affected this: v0.4.2...v0.5.0.

[ajschmidt8]

here is a better stacktrace:

[2023-06-07T18:47:09Z DEBUG sccache::config] Attempting to read config file at "/root/.config/sccache/config"
[2023-06-07T18:47:09Z DEBUG sccache::config] Couldn't open config file: failed to open file `/root/.config/sccache/config`
[2023-06-07T18:47:09Z DEBUG sccache::config] Attempting to read config file at "/root/.config/sccache/config"
[2023-06-07T18:47:09Z DEBUG sccache::config] Couldn't open config file: failed to open file `/root/.config/sccache/config`
[2023-06-07T18:47:09Z INFO  sccache::server] start_server: port: 4226
[2023-06-07T18:47:09Z INFO  sccache::server] No scheduler address configured, disabling distributed sccache
[2023-06-07T18:47:09Z DEBUG sccache::cache::cache] Init s3 cache with bucket <redacted_bucket>, endpoint None
[2023-06-07T18:47:09Z DEBUG opendal::services::s3::backend] backend build started: Builder { root: Some("libcudf-linux64/"), bucket: "<redacted_bucket>", endpoint: None, region: Some("us-east-2"), .. }
[2023-06-07T18:47:09Z DEBUG opendal::services::s3::backend] backend use root /libcudf-linux64/
[2023-06-07T18:47:09Z DEBUG opendal::services::s3::backend] backend use bucket <redacted_bucket>
[2023-06-07T18:47:09Z DEBUG opendal::services::s3::backend] backend use region: us-east-2
[2023-06-07T18:47:09Z DEBUG opendal::services::s3::backend] backend use endpoint: https://s3.us-east-2.amazonaws.com/<redacted_bucket>
[2023-06-07T18:47:09Z DEBUG opendal::services::s3::backend] backend build finished
[2023-06-07T18:47:09Z DEBUG opendal::services] service=s3 operation=metadata -> started
[2023-06-07T18:47:09Z DEBUG opendal::services] service=s3 operation=metadata -> finished: AccessorInfo { scheme: S3, root: "/libcudf-linux64/", name: "<redacted_bucket>", capability: { Stat | Read | Write | CreateDir | Delete | Copy | List | Presign | Batch } }
[2023-06-07T18:47:09Z DEBUG opendal::services] service=s3 operation=read path=.sccache_check range=0- -> started
[2023-06-07T18:47:09Z DEBUG reqwest::connect] starting new connection: http://169.254.169.254/
[2023-06-07T18:47:09Z DEBUG hyper::client::connect::http] connecting to 169.254.169.254:80
sccache: error: Timed out waiting for server startup. Maybe the remote service is unreachable?
Run with SCCACHE_LOG=debug SCCACHE_NO_DAEMON=1 to get more information

The 169.254.169.254 IP address is mentioned in the stacktrace, which is the endpoint for retrieve EC2 instance metdata (src: aws docs).

So it seems like it could be related to the disable_ec2_metadata opendal changes described here: https://github.com/apache/incubator-opendal/blob/91611ca88b08a41daa4f2a64d573c3fbaea1fdd3/core/src/docs/upgrade.md?plain=1#L55-L57

It looks like it's trying to connect to that 169.254.169.254 endpoint on a machine that isn't an EC2 instance.

[Xuanwo]

Thank you for investigating. The fix has been submitted.