Upload files failing after approximately 50 files

[abdelali-hamza]

Describe the bug

When uploading multiple files (approximately 50 ) to a table via the API or manually, the upload process starts failing with HTTPConnectionPool(host='127.0.0.1', port=3000): Read timed out. (read timeout=10) errors. This issue occurs consistently after processing around 50 files, regardless of whether the uploads are performed via a script or manually. The server appears to handle the initial uploads successfully but begins timing out on subsequent requests.

Config for the upload

Using s3 uploads

the file is uploaded to the bucket successfully , i can find it when i browse data

Screenshots

2025-03-10 05:19:26,471 - ERROR - Failed to upload file for record recbhV4sVwe5zvLAdvO: HTTPConnectionPool(host='127.0.0.1', port=3000): Read timed out. (read timeout=10)

logs of the container :

{
"level": 30,
"time": 1741585801380,
"pid": 6,
"hostname": "304fad8d44ea",
"name": "teable",
"req": {
"id": "a0e28219853287547357c4ad1293a7c9",
"method": "POST",
"url": "/api/table/tblHXuC0uTgITgxVEYO/record/recBF5jpyCvPdSQpO8g/fldMZLtpoIonS5S4Yiz/uploadAttachment",
"query": {},
"params": {
"0": "api/table/tblHXuC0uTgITgxVEYO/record/recBF5jpyCvPdSQpO8g/fldMZLtpoIonS5S4Yiz/uploadAttachment"
},
"remoteAddress": "::ffff:192.168.65.1",
"remotePort": 47868
},
"context": "AttachmentsService",
"spanId": "239164258fc5138f",
"traceId": "a0e28219853287547357c4ad1293a7c9",
"msg": "Uploading file: HHLmyvL1GFlC.pdf, size: 186853 bytes, mimetype: application/pdf"
}

{
"level": 30,
"time": 1741585831334,
"pid": 6,
"hostname": "304fad8d44ea",
"name": "teable",
"req": {
"id": "a0e28219853287547357c4ad1293a7c9",
"method": "POST",
"url": "/api/table/tblHXuC0uTgITgxVEYO/record/recBF5jpyCvPdSQpO8g/fldMZLtpoIonS5S4Yiz/uploadAttachment",
"query": {},
"params": {
"0": "api/table/tblHXuC0uTgITgxVEYO/record/recBF5jpyCvPdSQpO8g/fldMZLtpoIonS5S4Yiz/uploadAttachment"
},
"remoteAddress": "::ffff:192.168.65.1",
"remotePort": 47868
},
"res": {
"statusCode": null
},
"responseTime": 30004,
"spanId": "239164258fc5138f",
"traceId": "a0e28219853287547357c4ad1293a7c9",
"msg": "request aborted"
}

when i try to upload , it stays in this step

it seems like the request http://localhost:3000/api/attachments/notify/QbzzQEJQIYNr?filename=C_Diapo.pdf is stuck

Temporary Solution

docker compose down
docker compose up -d

Additional context

The issue persists even with manual uploads, suggesting a server-side limitation rather than a client-side script problem.
The backend code (AttachmentsService) and thresholdConfig do not impose an explicit limit on the number of uploads, only file size limits (maxAttachmentUploadSize and maxOpenapiAttachmentUploadSize, both set to Infinity by default).
Logs indicate successful record creation but fail during attachment upload with a 10-second timeout (set in the script and possibly reflected in server behavior).
Server logs or configuration details (e.g., Prisma connection pool, NestJS timeout settings) might provide further insight.
Example log snippet:
2025-03-10 05:19:26,471 - ERROR - Failed to upload file for record recbhV4sVwe5zvLAdvO: HTTPConnectionPool(host='127.0.0.1', port=3000): Read timed out. (read timeout=10)

[dataskis]

Update on this one? Daily complaints & resulting restart is a habit we'd love to break. I'm not sure its a great idea to automate healthcheck and self-healing given how happy-path it is. Looks like an Nginx config/conflict issue from what we're looking at (at first glance), so we're exploring that area. Hope to receive more definitive guidance here given that this is a fundamental issue.

Also, would using the MinIO locally instead of S3 help maybe? Or May give that a try.

[tea-artist]

We have noticed that there are certain issues with the support for s3 and will find time to fix them in the near future

[boris-w]

Hello! Are you still experiencing the same error? Or have you tried switching to Minio as an alternative storage solution?

[gnpaone]

No, it's the same thing with minio, I face the same issue. Can I try to solve the issue?

[boris-w]

No, it's the same thing with minio, I face the same issue. Can I try to solve the issue?

I'm very glad to hear that, if you can.

[gnpaone]

I found the issue. As large data is uploaded the ram consumption jumps considerably and when ram is full the request is getting aborted. I think while uploading pdf data is stored in ram and stays in ram until page reload or browser reopen or docker restart. Opening pdf from table works fine as it's just pre-signed url. I am not sure but I think this is what's happening.

[gnpaone]

By the way I am using external minio. There's some minor mistakes with documentation as well as code and I will submit a PR soon for that.

[benzfy]

I encountered the same issue. The upload gets stuck at 100%.
I noticed that the POST request to api/attachments/notify remains in PENDING status.

[benzfy]

我最终发现后端是卡在了notify的处理上。

通过打印日志我发现了一些端倪

增加日志的代码：

  async notify(token: string, filename?: string): Promise<INotifyVo> {
    this.logger.log(`[PERF] Notify开始处理 - Token: ${token}`);
    const totalStartTime = Date.now();
    const steps: {step: string; duration: number}[] = [];
    let stepStartTime = totalStartTime;

    // 1. 获取缓存中的token信息
    this.logger.log(`[PERF] Notify步骤1 - 开始获取缓存 - Token: ${token}`);
    const tokenCache = await this.cacheService.get(`attachment:signature:${token}`);
    const step1Duration = Date.now() - stepStartTime;
    steps.push({step: '获取缓存', duration: step1Duration});
    this.logger.log(`[PERF] Notify步骤1 - 获取缓存完成 - Token: ${token}, 耗时: ${step1Duration}ms`);
    stepStartTime = Date.now();

    if (!tokenCache) {
      throw new BadRequestException(`Invalid token: ${token}`);
    }
    const userId = this.cls.get('user.id');
    const { path, bucket } = tokenCache;

    // 2. 获取对象元数据
    this.logger.log(`[PERF] Notify步骤2 - 开始获取对象元数据 - Token: ${token}, Path: ${path}, Bucket: ${bucket}`);
    const { hash, size, mimetype, width, height, url } = await this.storageAdapter.getObjectMeta(
      bucket,
      path,
      token
    );
    const step2Duration = Date.now() - stepStartTime;
    steps.push({step: '获取对象元数据', duration: step2Duration});
    this.logger.log(`[PERF] Notify步骤2 - 获取对象元数据完成 - Token: ${token}, 耗时: ${step2Duration}ms, 文件大小: ${size}, 文件类型: ${mimetype}`);
    stepStartTime = Date.now();

    // 3. 创建数据库记录
    this.logger.log(`[PERF] Notify步骤3 - 开始创建数据库记录 - Token: ${token}`);
    const attachment = await this.prismaService.txClient().attachments.create({
      data: {
        hash,
        size,
        mimetype,
        token,
        path,
        width,
        height,
        createdBy: userId,
      },
      select: {
        token: true,
        size: true,
        mimetype: true,
        width: true,
        height: true,
        path: true,
      },
    });
    const step3Duration = Date.now() - stepStartTime;
    steps.push({step: '创建数据库记录', duration: step3Duration});
    this.logger.log(`[PERF] Notify步骤3 - 创建数据库记录完成 - Token: ${token}, 耗时: ${step3Duration}ms`);
    stepStartTime = Date.now();

    // 4. 添加裁剪任务到队列
    this.logger.log(`[PERF] Notify步骤4 - 开始添加裁剪任务到队列 - Token: ${token}`);
    await this.attachmentsCropQueueProcessor.queue.add('attachment_crop_image', {
      token: attachment.token,
      path: attachment.path,
      mimetype: attachment.mimetype,
      height: attachment.height,
      bucket,
    });
    const step4Duration = Date.now() - stepStartTime;
    steps.push({step: '添加裁剪任务到队列', duration: step4Duration});
    this.logger.log(`[PERF] Notify步骤4 - 添加裁剪任务到队列完成 - Token: ${token}, 耗时: ${step4Duration}ms`);
    stepStartTime = Date.now();

日志显示，主要是卡在了”开始获取对象元数据“这一步
一开始是有一些过慢的原数据获取

[13:32:24.471] INFO (teable/72328): [PERF] Notify完成处理 - Token: 0GRVCQgapGcH, 总耗时: 57304ms, 步骤耗时: [{"step":"获取缓存","duration":1},{"step":"获取对象元数据","duration":57289},{"step":"创建数据库记录","duration":6},{"step":"添加裁剪任务到队列","duration":0},{"step":"获取预览URL","duration":7}]

然后劣化到notify请求直接卡死

[13:33:10.913] INFO (teable/72328): [PERF] Notify开始处理 - Token: 94K1ZgD1yvxz
    req: {
      "id": "e80a30a0d931606feee789bfdeade9d2",
      "method": "POST",
      "url": "/api/attachments/notify/94K1ZgD1yvxz?filename=part_0_Rz2sA+(1).txt",
      "query": {
        "filename": "part_0_Rz2sA (1).txt"
      },
      "params": {
        "0": "api/attachments/notify/94K1ZgD1yvxz"
      },
      "remoteAddress": "::1",
      "remotePort": 53268
    }
    context: "AttachmentsService"
    spanId: "be86ffe506f40c8d"
    traceId: "e80a30a0d931606feee789bfdeade9d2"
[13:33:10.913] INFO (teable/72328): [PERF] Notify步骤1 - 开始获取缓存 - Token: 94K1ZgD1yvxz
    req: {
      "id": "e80a30a0d931606feee789bfdeade9d2",
      "method": "POST",
      "url": "/api/attachments/notify/94K1ZgD1yvxz?filename=part_0_Rz2sA+(1).txt",
      "query": {
        "filename": "part_0_Rz2sA (1).txt"
      },
      "params": {
        "0": "api/attachments/notify/94K1ZgD1yvxz"
      },
      "remoteAddress": "::1",
      "remotePort": 53268
    }
    context: "AttachmentsService"
    spanId: "be86ffe506f40c8d"
    traceId: "e80a30a0d931606feee789bfdeade9d2"
[13:33:10.913] INFO (teable/72328): [PERF] Notify步骤1 - 获取缓存完成 - Token: 94K1ZgD1yvxz, 耗时: 0ms
    req: {
      "id": "e80a30a0d931606feee789bfdeade9d2",
      "method": "POST",
      "url": "/api/attachments/notify/94K1ZgD1yvxz?filename=part_0_Rz2sA+(1).txt",
      "query": {
        "filename": "part_0_Rz2sA (1).txt"
      },
      "params": {
        "0": "api/attachments/notify/94K1ZgD1yvxz"
      },
      "remoteAddress": "::1",
      "remotePort": 53268
    }
    context: "AttachmentsService"
    spanId: "be86ffe506f40c8d"
    traceId: "e80a30a0d931606feee789bfdeade9d2"
[13:33:10.913] INFO (teable/72328): [PERF] Notify步骤2 - 开始获取对象元数据 - Token: 94K1ZgD1yvxz, Path: table/94K1ZgD1yvxz, Bucket: gil-test
    req: {
      "id": "e80a30a0d931606feee789bfdeade9d2",
      "method": "POST",
      "url": "/api/attachments/notify/94K1ZgD1yvxz?filename=part_0_Rz2sA+(1).txt",
      "query": {
        "filename": "part_0_Rz2sA (1).txt"
      },
      "params": {
        "0": "api/attachments/notify/94K1ZgD1yvxz"
      },
      "remoteAddress": "::1",
      "remotePort": 53268
    }
    context: "AttachmentsService"
    spanId: "be86ffe506f40c8d"
    traceId: "e80a30a0d931606feee789bfdeade9d2"

通过进一步分析，发现问题可能出现在获取元数据的时候会把下载下来。当遇到过大的附件的时候可能会出现非常慢的问题。

通过把下载替换成为获取Head可以解决这一问题

const command = new GetObjectCommand -> const headCommand = new HeadObjectCommand

代码修改为：

  async getObjectMeta(bucket: string, path: string): Promise<IObjectMeta> {
    const url = `/${bucket}/${path}`;
    // 使用 HeadObjectCommand 获取元数据，而不是 GetObjectCommand
    const headCommand = new HeadObjectCommand({
      Bucket: bucket,
      Key: path,
    });
    const {
      ContentLength: size,
      ContentType: mimetype,
      ETag: hash,
    } = await this.s3Client.send(headCommand);

    if (!size || !mimetype || !hash) {
      throw new BadRequestException('Invalid object meta from HeadObject');
    }

    if (!mimetype?.startsWith('image/')) {
      return {
        hash,
        size,
        mimetype,
        url,
      };
    }

    // 对于图片，仍然需要获取流来读取宽高信息
    // 注意：这里为了获取图片宽高，还是会下载图片内容。可以考虑后续优化为异步处理或其它方式。
    const getCommand = new GetObjectCommand({
      Bucket: bucket,
      Key: path,
    });
    const { Body: stream } = await this.s3Client.send(getCommand);

    if (!stream) {
      // 如果流获取失败，可以只返回基础元数据，或者抛出更具体的错误
      console.warn(`Failed to get image stream for ${path}, returning basic meta.`);
      return {
        hash,
        size,
        mimetype,
        url,
      };
    }

    const metaReader = sharp();
    const sharpReader = (stream as Readable).pipe(metaReader);
    // 添加错误处理，防止sharp处理失败导致整个请求失败
    try {
      const { width, height } = await sharpReader.metadata();
      return {
        hash,
        url,
        size,
        mimetype,
        width,
        height,
      };
    } catch (error) {
      console.error(`Error processing image metadata for ${path} with sharp:`, error);
      // 如果sharp处理失败，返回不含宽高的元数据
      return {
        hash,
        url,
        size,
        mimetype,
      };
    }
  }

经过初步测试，确实可以解决这个问题。

由于我对nextjs不是特别熟悉，这块的根因我还没找到，但似乎确实是修好了（初步测试）。
因为如果只是慢的话按理说等足够长的时间应该会好，但实际不会。一些猜测：可能是某些原因某些连接一直没被释放，最后S3Client 内部通常维护一个HTTP连接池，可能被耗尽了。

@boris-w
希望这个发现对于你们修复这个问题有帮助！
谢谢

[benzfy]

我找到根因了原因了，S3 Client有一个Related Issue:
S3 GetObjectCommand leaks sockets if the body is never read / add doc links for streaming responses re: socket exhaustion
aws/aws-sdk-js-v3#6691
所以这个修改是解决了这个问题。

[boris-w]

我找到根因了原因了，S3 Client有一个Related Issue: S3 GetObjectCommand leaks sockets if the body is never read / add doc links for streaming responses re: socket exhaustion aws/aws-sdk-js-v3#6691 所以这个修改是解决了这个问题。

Wow, excellent work! It does look quite credible. Minio has the same issue, but I checked Minio's code and it seems to be normal.
Feel free to submit a PR if you'd like! If not, no worries - I'll check it out this week.

Thanks again for your great work on this!

[benzfy]

Glad my analysis was helpful. Regarding minIO, I understand it might be a separate issue.

Unfortunately, I won't have time to submit a PR this week, so I appreciate you offering to look into it. Thanks