Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

form_data不能处理ctx.getFileStream()流 #3201

Closed
xiabingwu opened this issue Nov 15, 2018 · 15 comments

Comments

Projects
None yet
6 participants
@xiabingwu
Copy link

commented Nov 15, 2018

反馈个问题,代码结构大概是这样的,使用request包转发web页面上传过来的文件流
request.post({url:'服务接口', formData: {
'FileData':ctx.getFileStream()
}}, function optionalCallback(err, httpResponse, body) {
console.log(body);
});
然后服务接口返回的结果是
{"response":{"code":-1,"msg":"The uploaded file was only partially uploaded","cost":"1ms"},"data":[],"code":-1}
意思是文件只上传了一部分。
通过查阅request依赖的form-data包发现里面form_data.js文件FormData.prototype._trackLength方法有一段代码
// empty or either doesn't have path or not an http response
if (!value || ( !value.path && !(value.readable && value.hasOwnProperty('httpVersion')) )) {
return;
}
表示如果value为空,或者value不是来自文件或者http响应的流,就直接拒绝了
而通过ctx.getFileStream()获取的流的格式大概是这样的
FileStream {
_readableState:
ReadableState {
objectMode: false,
highWaterMark: 16384,
buffer: BufferList { head: [Object], tail: [Object], length: 1 },
length: 63522,
pipes: null,
pipesCount: 0,
flowing: null,
ended: false,
endEmitted: false,
reading: false,
sync: true,
needReadable: false,
emittedReadable: false,
readableListening: false,
resumeScheduled: false,
destroyed: false,
defaultEncoding: 'utf8',
awaitDrain: 0,
readingMore: false,
decoder: null,
encoding: null },
readable: true,
domain: null,
_events:
{ end: [Function],
limit: { [Function: bound onceWrapper] listener: [Function] } },
_eventsCount: 2,
_maxListeners: undefined,
truncated: false,
_read: [Function],
fieldname: 'Filedata',
filename: '3.jpg',
encoding: '7bit',
transferEncoding: '7bit',
mime: 'image/jpeg',
mimeType: 'image/jpeg',
fields:
{ id: 'WU_FILE_13',
name: '3.jpg',
type: 'image/jpeg',
lastModifiedDate: 'Thu Oct 11 2018 20:14:26 GMT+0800 (中国标准时间)',
appkey: '1',
isRetImgAttr: '1',
from: 'user' } }
这样被form-data认为既不是一个文件流也不是一个http响应流,导致后期content-length计算错误

@atian25

This comment has been minimized.

Copy link
Member

commented Nov 15, 2018

这样贴代码都不带 ``` 格式化的,很难看的。

按要求提供下最小可复现代码库。

PS:如果你不熟悉 stream 的话,直接用 file mode。

@popomore

This comment has been minimized.

Copy link
Member

commented Nov 16, 2018

如果做转发的话把你的中间件放最前面,直接 pipe request

@xiabingwu

This comment has been minimized.

Copy link
Author

commented Nov 17, 2018

@atian25 非常抱歉,代码没有按照规范来。最小可复现代码如下:

    const stream = await ctx.getFileStream();
    ctx.body=request.post({
       url:'http://127.0.0.1:7001/saveFile',
       formData:{
         FileData:stream
    }})

获取文件流然后只转发文件流会失败,怎样才能做到只转发文件流成功呢
但直接转发req流可行

ctx.body=ctx.req.pipe(request.post({ url: 'http://127.0.0.1:7001/saveFile' }));

谢谢 @popomore 提醒

@popomore

This comment has been minimized.

Copy link
Member

commented Nov 17, 2018

自己排查吧,可能你使用 request 的姿势不对。

@popomore popomore closed this Nov 17, 2018

@xiabingwu

This comment has been minimized.

Copy link
Author

commented Nov 19, 2018

自己排查吧,可能你使用 request 的姿势不对。

我这里搞了个demo,放这里了https://github.com/xiabingwu/upload-demo
,没有业务,只把文件流转发的另外一个接口,转发不过去
能帮忙看看嘛,主要代码在这里
https://github.com/xiabingwu/upload-demo/blob/master/app/controller/upload.js
@atian25 @popomore

@atian25

This comment has been minimized.

Copy link
Member

commented Nov 19, 2018

request 我不熟,你可以试下 https://www.npmjs.com/package/urllib#options-optionsstream

egg 自带的 httpclient 就是它

@longhaiyan

This comment has been minimized.

Copy link

commented May 8, 2019

@xiabingwu
找到解决方案啦,可直接使用 fileStream 进行上传。亲测可用~

首先:
如图,用 request 模拟 multipart/form-data 发送文件时,如果直接用 eggfileStream 的话,发送时文件数据是不全的,原因可能真的是楼主开头所说的...
image

其次:
现在常见的解决方案有,将 fileStream 写到文件中,再用 createReadStreamstream 来上传。
感觉不友好,挣扎着找其它出路...

最后:
既然 request 不支持使用 fileStream ,那就换个工具吧,试试 unirest

const unirest = require('unirest');
const { ctx } = this;
const fileStream = await ctx.getFileStream();

const req = unirest('POST', 'http://xx.cn/upload');

req.headers({
  'cache-control': 'no-cache',
  'content-type': 'multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW',
});

req.multipart([
  {
    'Content-Disposition': 'form-data;name="title"',
    body: '这是非文件数据',
  },
  {
    'Content-Disposition': 'form-data;name="uploadFile";filename="tmp.jpg"',
    'Content-Type': 'image/jpeg', // 具体的 Content-Type 可以从 fileStream 里面拿
    body: fileStream, 
  },
]);

req.end(function(res) {
  if (res.error) throw new Error(res.error);
  console.log(res.body);
});

注意: 使用前不要消费掉 fileStream,如果需要消费,那最好先用 Stream.PassThrough() 复制一份

@fengmk2

This comment has been minimized.

Copy link
Member

commented May 8, 2019

建议对 stream 不熟悉的同学还是直接使用默认的文件模式吧,简单易懂 https://eggjs.org/zh-cn/basics/controller.html#file-%E6%A8%A1%E5%BC%8F

@longhaiyan

This comment has been minimized.

Copy link

commented May 9, 2019

建议对 stream 不熟悉的同学还是直接使用默认的文件模式吧,简单易懂 https://eggjs.org/zh-cn/basics/controller.html#file-%E6%A8%A1%E5%BC%8F

在 stream 上撞墙的,估计都改 file 了吧,仍挣扎于 stream 的,应该是没得选

@fengmk2

This comment has been minimized.

Copy link
Member

commented May 9, 2019

@longhaiyan 你可以提供可以复现的代码库,说不定我们能发现问题所在。

@waitingsong

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

我这儿测试读取本地文件然后发送,也是相同的问题

文档: https://eggjs.org/api/Context.html#getFileStream
getFileStream(options) → {ReadStream}

实际上返回类型是 FileStram,而不是 ReadStrem, 当然主要问题不在这儿。

@waitingsong

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

下图, stream 变量是 Filestream, stream2 是 ReadStream

2019-05-09_180657

备注: 上图的 readable 值是因为写入文件后改变为 false 的。

2019-05-09_180759

waitingsong added a commit to waitingsong/upload-demo that referenced this issue May 9, 2019

waitingsong added a commit to waitingsong/upload-demo that referenced this issue May 9, 2019

@waitingsong

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

原因:

ctx.getFileStream() 只能处理通过表单上传的文件(流), 而转发过来的 ReadStream 是没有 Content-Type 之类键名,所以 ctx.getFileStream() 调用到的 multipart() 方法检查到 Content-Type 值非 Content-Type must be multipart/* 就会抛出异常。

解决方案:

https://github.com/waitingsong/upload-demo

  • 直接发送数据流,而非采用 FormData 构造表单文件域
  • 在 saveFile 接口中直接读取数据流,而非使用 ctx.getFileStream() 来处理

不清楚 request 模块如何发送原始数据流,所以代码用的自己的轮子。可自行尝试 request

waitingsong added a commit to waitingsong/upload-demo that referenced this issue May 9, 2019

waitingsong added a commit to waitingsong/upload-demo that referenced this issue May 9, 2019

@longhaiyan

This comment has been minimized.

Copy link

commented May 10, 2019

@waitingsong

  • egg 官方说,FileStram 是 ReadStrem 的子类,所以 FileStram 是可以当做 ReadStrem 来用的
  • 这个 issues 的楼主遇到的问题应该是「egg 中使用 request 包转发 web 页面上传过来的文件流失败」吧,所以文件流是来自表单上传的。你提到 「转发过来的 ReadStream ... 就会抛出异常」跟本 issues 好像不是同一个问题。
@waitingsong

This comment has been minimized.

Copy link
Contributor

commented May 10, 2019

@longhaiyan 我是这样理解的: egg 接收到浏览器表单提交上来的文件(流),然后直接转发到 save 接口时save 接口调用 ctx.getFileStream() 方法会异常。
难道搞错了……?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.