Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Help] s3 使用 contabo 失败 #64

Closed
1PercentSync opened this issue Jun 8, 2024 · 15 comments · Fixed by #67
Closed

[Help] s3 使用 contabo 失败 #64

1PercentSync opened this issue Jun 8, 2024 · 15 comments · Fixed by #67
Labels
good for first contribution Want to contribute to Rin? Start here! help wanted Extra attention is needed PR welcome Good for new contributor

Comments

@1PercentSync
Copy link
Contributor

这家在其他软件中需要选中强制路径样式，是否和这个有关系
@1PercentSync 1PercentSync added the help wanted Extra attention is needed label Jun 8, 2024
@1PercentSync
Copy link
Contributor Author

image

@1PercentSync
Copy link
Contributor Author

1PercentSync commented Jun 8, 2024 •
edited by OXeu
Loading

后端log

{
  "outcome": "ok",
  "scriptVersion": {
    "id": "a60eabfa-45dd-499e-b3e5-53e489b48eb2"
  },
  "scriptName": "rin-server",
  "diagnosticsChannelEvents": [],
  "exceptions": [],
  "logs": [
    {
      "message": [
        "char 'e' is not expected.:1:1\n  Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object."
      ],
      "level": "error",
      "timestamp": 1717819383229
    }
  ],
  "eventTimestamp": 1717819383200,
  "event": {
    "request": {
      "url": "https://rin-server.1percentsync.games/storage",
      "method": "POST",
      "headers": {
        "accept": "*/*",
        "accept-encoding": "gzip, br",
        "accept-language": "zh-CN,zh-TW;q=0.9,zh;q=0.8,en-US;q=0.7,en;q=0.6",
        "authorization": "REDACTED",
        "cf-connecting-ip": "2601:19c:5000:120:d92b:b6d5:bc89:7fae",
        "cf-ipcountry": "US",
        "cf-ray": "8905fd66cc2e9e1a",
        "cf-visitor": "{\"scheme\":\"https\"}",
        "connection": "Keep-Alive",
        "content-length": "1865858",
        "content-type": "multipart/form-data; boundary=----WebKitFormBoundaryfIfBdjZv9QozZbOz",
        "dnt": "1",
        "host": "rin-server.1percentsync.games",
        "origin": "https://rin.1percentsync.games",
        "priority": "u=1, i",
        "referer": "https://rin.1percentsync.games/",
        "sec-ch-ua": "\"Microsoft Edge\";v=\"125\", \"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\"",
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": "\"Windows\"",
        "sec-fetch-dest": "empty",
        "sec-fetch-mode": "cors",
        "sec-fetch-site": "same-site",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0",
        "x-forwarded-proto": "https",
        "x-real-ip": "2601:19c:5000:120:d92b:b6d5:bc89:7fae"
      },
      "cf": {
        "longitude": "-71.19840",
        "httpProtocol": "HTTP/3",
        "tlsCipher": "AEAD-AES128-GCM-SHA256",
        "continent": "NA",
        "asn": 7922,
        "clientAcceptEncoding": "gzip, deflate, br, zstd",
        "country": "US",
        "verifiedBotCategory": "",
        "tlsClientAuth": {
          "certIssuerDNLegacy": "",
          "certIssuerSKI": "",
          "certSubjectDNRFC2253": "",
          "certSubjectDNLegacy": "",
          "certFingerprintSHA256": "",
          "certNotBefore": "",
          "certSKI": "",
          "certSerial": "",
          "certIssuerDN": "",
          "certVerified": "NONE",
          "certNotAfter": "",
          "certSubjectDN": "",
          "certPresented": "0",
          "certRevoked": "0",
          "certIssuerSerial": "",
          "certIssuerDNRFC2253": "",
          "certFingerprintSHA1": ""
        },
        "tlsExportedAuthenticator": {
          "clientFinished": "fbbf24a818ea0b409eef0839b331f7f3bb902b7139b4cf4f1c51fda6fcb67b8b",
          "clientHandshake": "abdc5186fac35c791a78fa65ff2030401432028672dc2c8cfcd5d79b271e15ff",
          "serverHandshake": "8e9c977f06ee9cf272f8100e0fd5a54104716f64705feee9b8b171cae0a723a1",
          "serverFinished": "255c8fd35d1f609dca0f2355a4b5b8b91ee23e5b6a648eaae884399e155ebacd"
        },
        "tlsVersion": "TLSv1.3",
        "city": "Burlington",
        "timezone": "America/New_York",
        "colo": "EWR",
        "tlsClientHelloLength": "2075",
        "edgeRequestKeepAliveStatus": 1,
        "postalCode": "01803",
        "region": "Massachusetts",
        "latitude": "42.50900",
        "requestPriority": "",
        "regionCode": "MA",
        "asOrganization": "Comcast Cable",
        "metroCode": "506",
        "tlsClientExtensionsSha1": "QgXfefoqdcH2v14yfHZiKrQykVs=",
        "tlsClientRandom": "jxiiDr9bE7oOsbcPuwa4QsFKcv60WSQnt0qeNqwGjJ4=",
        "botManagement": {
          "corporateProxy": false,
          "verifiedBot": false,
          "jsDetection": {
            "passed": false
          },
          "staticResource": false,
          "detectionIds": {},
          "score": 99
        }
      }
    },
    "response": {
      "status": 400
    }
  },
  "id": 1
}

@1PercentSync
Copy link
Contributor Author

需要contabo的s3测试的话我可以提供

@OXeu OXeu changed the title [Help] s3使用contabo失败 [Help] s3 使用 corntab 失败 Jun 8, 2024
@1PercentSync
Copy link
Contributor Author

nope,就叫https://contabo.com/

@OXeu
Copy link
Collaborator

OXeu commented Jun 8, 2024

nope,就叫https://contabo.com/

抱歉😂

@OXeu OXeu changed the title [Help] s3 使用 corntab 失败 [Help] s3 使用失败 Jun 8, 2024
@OXeu OXeu changed the title [Help] s3 使用失败 [Help] s3 使用 contabo 失败 Jun 8, 2024
@1PercentSync
Copy link
Contributor Author

测试了backblaze是没问题的

@OXeu
Copy link
Collaborator

OXeu commented Jun 8, 2024

没有使用过这家的服务，晚点我去看看他们的文档

@1PercentSync
Copy link
Contributor Author

没有使用过这家的服务，晚点我去看看他们的文档

来自gpt补充的可能相关信息：
在某些情况下，需要选中S3的强制路径样式（Path Style）是因为路径样式在特定的网络配置或旧版S3 API兼容性问题中能够确保更好的兼容性和连接成功。主要原因包括：

  1. DNS解析问题：某些网络环境下，对象存储的虚拟主机（Virtual Host）样式的DNS解析可能会失败，而路径样式不依赖于DNS解析，因此可以避免这些问题。

  2. 兼容性：一些旧版的S3 SDK或工具可能不支持虚拟主机样式，只能使用路径样式进行访问。

  3. 跨区域访问：在某些跨区域访问的场景中，路径样式可能会比虚拟主机样式更稳定。

具体情况下，选择路径样式还是虚拟主机样式应根据实际需求和环境来决定。

@1PercentSync
Copy link
Contributor Author

awslabs/aws-sdk-rust#390 (comment)

@OXeu
Copy link
Collaborator

OXeu commented Jun 8, 2024

你可以先在自己的仓库中修改 server/src/services/storage.ts 中
https://github.com/OXeu/Rin/blob/9b23ab6e09ba3d9149e06b3d6e4875c0c4fc3f57/server/src/services/storage.ts#L22-L29

添加一行参数forcePathStyle: true,如下所示：

    const s3 = new S3Client({
        region: region,
        endpoint: endpoint,
        forcePathStyle: true,
        credentials: {
            accessKeyId: accessKeyId,
            secretAccessKey: secretAccessKey,
        }
    });

然后测试是否能够正常工作，如果可以的话我后续加一个环境变量来实现这个

@OXeu
Copy link
Collaborator

OXeu commented Jun 8, 2024

你可以先在自己的仓库中修改 server/src/services/storage.ts 中

https://github.com/OXeu/Rin/blob/9b23ab6e09ba3d9149e06b3d6e4875c0c4fc3f57/server/src/services/storage.ts#L22-L29

添加一行参数forcePathStyle: true,如下所示：

    const s3 = new S3Client({
        region: region,
        endpoint: endpoint,
        forcePathStyle: true,
        credentials: {
            accessKeyId: accessKeyId,
            secretAccessKey: secretAccessKey,
        }
    });

然后测试是否能够正常工作，如果可以的话我后续加一个环境变量来实现这个

同时也欢迎 PR

@OXeu OXeu added PR welcome Good for new contributor good for first contribution Want to contribute to Rin? Start here! labels Jun 8, 2024
@1PercentSync
Copy link
Contributor Author

可以，成功了，我研究研究怎么整环境变量

@1PercentSync
Copy link
Contributor Author

有个问题：
我修改了两处，一个是
storage.ts:
export const StorageService = (db: DB, env: Env) => {
const region = env.S3_REGION;
const endpoint = env.S3_ENDPOINT;
const accessKeyId = env.S3_ACCESS_KEY_ID;
const secretAccessKey = env.S3_SECRET_ACCESS_KEY;
const accessHost = env.S3_ACCESS_HOST || endpoint;
const bucket = env.S3_BUCKET;
const folder = env.S3_FOLDER || '';
const forcePathStyle = env.FORCE_PATH_STYLE || false; //添加了FORCE_PATH_STYLE
const s3 = new S3Client({
region: region,
endpoint: endpoint,
forcePathStyle: forcePathStyle,
credentials: {
accessKeyId: accessKeyId,
secretAccessKey: secretAccessKey
}
});

然后在db.ts中添加了：
export interface Env {
DB: D1Database;
RIN_GITHUB_CLIENT_ID: string;
RIN_GITHUB_CLIENT_SECRET: string;
GITHUB_CLIENT_ID: string;
GITHUB_CLIENT_SECRET: string;
JWT_SECRET: string;
FRONTEND_URL: string;
S3_REGION: string,
S3_ENDPOINT: string,
S3_ACCESS_KEY_ID: string,
S3_SECRET_ACCESS_KEY: string,
S3_ACCESS_HOST: string,
S3_BUCKET: string,
S3_FOLDER: string,
S3_CACHE_FOLDER: string,
WEBHOOK_URL: string,
FORCE_PATH_STYLE: boolean, //这一行
}
然而配置完环境变量部署是不生效的

@1PercentSync
Copy link
Contributor Author

环境变量是FORCE_PATH_STYLE=true

@OXeu
Copy link
Collaborator

OXeu commented Jun 8, 2024 •
edited
Loading

  • 添加的环境变量应以 S3_ 开头以便于确认用途，推荐使用 S3_FORCE_PATH_STYLE 作为环境变量名称
  • 添加一个新的环境变量需要修改以下地方：

  1. 在使用处添加环境变量的引用变量并根据情况配置合适的默认值，此处建议：https://github.com/OXeu/Rin/blob/9b23ab6e09ba3d9149e06b3d6e4875c0c4fc3f57/server/src/services/storage.ts#L15-L21

    const forcePathStyle = env.S3_FORCE_PATH_STYLE === "1" || false;

  2. 在 Env 类型处声明对应的环境变量，类型统一为 string（使用 string 而不是 boolean 防止出现错误的类型转换导致意料之外的结果）：
    https://github.com/OXeu/Rin/blob/9b23ab6e09ba3d9149e06b3d6e4875c0c4fc3f57/server/src/db/db.ts#L3-L20

  3. 在部署的脚本 scripts/migrator.ts 使用并注入环境变量，明文需要放置到生成 wrangler.toml 的样板文本中，密文需在 Put secrets 流程处执行添加
    https://github.com/OXeu/Rin/blob/9b23ab6e09ba3d9149e06b3d6e4875c0c4fc3f57/scripts/migrator.ts#L21-L26
    https://github.com/OXeu/Rin/blob/9b23ab6e09ba3d9149e06b3d6e4875c0c4fc3f57/scripts/migrator.ts#L47-L53
    https://github.com/OXeu/Rin/blob/9b23ab6e09ba3d9149e06b3d6e4875c0c4fc3f57/scripts/migrator.ts#L112-L129

  4. 最后在 CI 文件 .github/workflows/deploy.yaml 中为其注入在 Github 中配置好的环境变量：
    https://github.com/OXeu/Rin/blob/9b23ab6e09ba3d9149e06b3d6e4875c0c4fc3f57/.github/workflows/deploy.yaml#L40-L45

同时，由于 SEO 也需要使用 S3 存储，你还需要修改 srcipts/render.ts 和 .github/workflows/seo.yaml 确保 SEO 也能正常运行
https://github.com/OXeu/Rin/blob/9b23ab6e09ba3d9149e06b3d6e4875c0c4fc3f57/scripts/render.ts#L8-L14
https://github.com/OXeu/Rin/blob/9b23ab6e09ba3d9149e06b3d6e4875c0c4fc3f57/.github/workflows/seo.yaml#L35-L43

@OXeu OXeu mentioned this issue Jun 8, 2024
Merged
@OXeu OXeu closed this as completed Jun 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good for first contribution Want to contribute to Rin? Start here! help wanted Extra attention is needed PR welcome Good for new contributor
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add S3_FORCE_PATH_STYLE 1PercentSync/Rin
2 participants
@OXeu @1PercentSync