Unknown stream handling (content-length unknown)

[derjust]

Hi everyone,

This is more a documentation/enhancement discussion than anything else:

My use case was to send a ReadableStream via multipart/form-encoded (more or less a form with just a file input field) to a remote host. The remote side was 'fxied' and I can't do anything about it.
Without the desire to create temporary files the challenge was to stream the input directly to the remote host.

This turned out to be way more complicated as the length of the input stream is unknown. Whereas with chunking it is absolutely fine to have no content-length.

Request on it's own tries very hard to come up with a content-length whereas the FormData.getLength calculates the wrong length (in fact counting the boundary string length only).

With the _lengthRetrievers there is a nice approach to define the length for streams. And throwing in error causes Request.js to skip the content-length altogether.

      self._form.getLength(function (err, length) {
        if (!err) {
          self.setHeader('content-length', length)
        }

with in turn also activates chunking which is working as expected.

But to get into this handling, the value must be specially crafted. As the value has to look like a file stream as there are two different validations: Fullfilling this

if (!value || ( !value.path && !(value.readable && value.hasOwnProperty('httpVersion')) )) {

allows access for the lazy length determination.
Whereas the logic later on tests for this: value.hasOwnProperty('fd')

Therefore with this kind of code:

    var pipedStream = getMyReadableStream();
    var metadata = getMyMetaDataForReadableStream();
    pipedStream.path = metadata.fileName; //This fools the logic and activates chunking

    var formData = {
        document: {
            value: pipedStream,
            options: {
                filename: metadata.fileName,
                contentType: metadata.contentType ? metadata.contentType : 'application/octet-stream'
            }
        }
    };
request.post({url:uploadUrl, formData: formData}, callback);

Therefore wouldn't it make sense to have an additional option which provides cleaner access to "We don't known how long it is"?

[derjust]

Maybe something like this (more like an idea, tests are missing etc): derjust@c2804ff

[thomas-riccardi]

This turned out to be way more complicated as the length of the input stream is unknown. Whereas with chunking it is absolutely fine to have no content-length.

And in fact it's forbidden by the HTTP spec: it's either content-lenght or content-encoding, not both.

See #70 where support for streams of unknown size is proposed.

[formula1]

Another Related PR here #70

This will be pretty important for me. It would not be ideal making a dependency on a fork, opening up multiple connections or creating a protocol for this : /

[alexindigo]

It will be solved in 1.0 as proposed in #70.