FormatException with non-ASCII header field value in request

[leutb]

Field value containing a non English character is invalid like 'Pétange'.
If changed to 'Petange' is OK.
Any idea ?

Map<String, String> _headers = { 'authorization': basicAuth, 'Entity': 'Pétange', }; http.Response _result = await _webUtils.get(_url, headers: _headers).then((dynamic res) { return res; });

Error :
FormatException: Invalid HTTP header field value: "Pétange"

[natebosch]

This looks like a bug around here:

http/lib/src/io_client.dart

Line 42 in 45f4082

ioRequest.headers.set(name, value);

I think we need something like:

final headerEncoding = Encoding.getByName('iso-8859-1');
request.headers.forEach((name, value) {
    ioRequest.headers.set(name, headerEncoding.encode(value));
});

I can't reproduce this on the web though, I'm not sure if string encoding is not something we need to worry about there...

[natebosch]

cc @mit-mit to triage.

I'm not sure if the SDK should be handling this for us, or if we need to handle it here... I'm also not super confident that my proposed solution would do the right thing.

[mit-mit]

cc @jonasfj wdyt?

[jonasfj]

Note RFC 7230. section 3.2 says: (emphasis added)

Historically, HTTP has allowed field content with text in the
ISO-8859-1 charset [ISO-8859-1], supporting other charsets only
through use of [RFC2047] encoding. In practice, most HTTP header
field values use only a subset of the US-ASCII charset [USASCII].
Newly defined header fields SHOULD limit their field values to
US-ASCII octets. A recipient SHOULD treat other octets in field
content (obs-text) as opaque data.

It seems to me that the inability to communicate with legacy systems that requires encoded header values could be considered a bug or a feature request.

Ideally, you should have to jump through some hoops to use non-ASCII header values, but it seems reasonable that this should be possible. Perhaps we can offload the encoding to the caller, I don't know how often anyone will need this.

---

@leutb, unless you're communicating with a legacy system I would suggest the string be UTF-8 and base64 encoded before sending it as a header value.

[natebosch]

Perhaps we can offload the encoding to the caller, I don't know how often anyone will need this.

We'd need to change the headers from Map<String, String> to Map<String, dynamic> to allow the value to be a List<int>. That might make the package harder to use.

[jonasfj]

Or add a new option rawHeaders with type Map<String, List<int>>.

I'm not sure there is an obvious fix..

[mattetti]

if you the info device in the user agent you now end up with the same issue:

Apple iPhone Xʀ iOS/12.2 Simulator which throws the same error.

[ghost]

Hello there,

I'm having a very similar issue. I'm making a GET request to an API, however it seems that it is returning an unsupported value in the response header:

[ERROR:flutter/lib/ui/ui_dart_state.cc(157)] Unhandled Exception: Invalid HTTP header field value: "273.756µs"
E/flutter ( 6704): #0      IOClient.send                   package:http/src/io_client.dart:65
E/flutter ( 6704): #1      BaseClient._sendUnstreamed      package:http/src/base_client.dart:176
E/flutter ( 6704): #2      BaseClient.get 

Is there any way this can be resolved? On my native app I am calling the same API using Retrofit and I am not having those issues...

[nilsreichardt]

Hello there,

I'm having a very similar issue. I'm making a GET request to an API, however it seems that it is returning an unsupported value in the response header:

[ERROR:flutter/lib/ui/ui_dart_state.cc(157)] Unhandled Exception: Invalid HTTP header field value: "273.756µs"
E/flutter ( 6704): #0      IOClient.send                   package:http/src/io_client.dart:65
E/flutter ( 6704): #1      BaseClient._sendUnstreamed      package:http/src/base_client.dart:176
E/flutter ( 6704): #2      BaseClient.get 

Is there any way this can be resolved? On my native app I am calling the same API using Retrofit and I am not having those issues...

Works with flutter stable? flutter channel stable

[ghost]

@AndroidNils Thanks for the quick reply. I've just tested it and indeed it works; on beta and dev it doesn't. I'm not sure though, if that means that in the next stable release this will also stop working on the stable channel...

[nilsreichardt]

@bnxm I'm facing with this issue too.

This code works only on stable...

import 'package:test/test.dart';
import 'package:http/http.dart' as http;

void main() {
  test('http', () async {
    const url = 'https://firebasestorage.googleapis.com/v0/b/sharezone-debug.appspot.com/o/files%2FVF1cIvA8bTebtmwqgVIf%2FQblBJKim2C6QIeJ8E8j6?alt=media&token=b712b0b6-9a63-4ae5-b9b2-cd0c2e110ece';
    var httpClient = http.Client();
    await httpClient.get(Uri.parse(url));
  });
}

Important: Metadata & ContentDisposition of this file includes german umlauts like ä, ö, ü .
Error: Invalid HTTP header field value: "attachment; filename="läl.jpg""
Workaround: Remove all german umlauts from metadata and contentDisposition

If I use the dart:io http client instead of http package, everything works (stable, beta and dev).

Same issue with Dio Package (https://pub.dev/packages/dio)

[ghost]

@AndroidNils I've filed an issue for this over here: dart-lang/sdk#41688

[natebosch]

@sortie - can you help me understand if the fix in the SDK covers this entire issue, or if there was some secondary issue that cropped up here and was solved? The SDK issue sounds like a recent regression but this was filed a while ago, and I can't quite tell from the discussion if there are multiple root causes.

[sortie]

@natebosch The SDK decodes HTTP headers as ISO-8859-1. I don't understand quite understand this question or what to do here. There was a regression for a while, introduced in https://dart-review.googlesource.com/c/sdk/+/138860 and fixed in https://dart-review.googlesource.com/c/sdk/+/145244. Is this issue still relevant? Do you have any specific question about the dart:io behavior, or an example of a problem, or something?

[natebosch]

The remaining question, is should the following code work, or should it throw an error:

import 'dart:io';

void main() async {
  var client = HttpClient();
  var request = await client.openUrl('GET', Uri.http('google.com', ''));
  request.headers.set('something', 'Pétange');
}

Running this results in:

Unhandled exception:
FormatException: Invalid HTTP header field value: "Pétange" (at character 2)
Pétange
 ^

#0      _HttpHeaders._validateValue (dart:_http/http_headers.dart:638:9)
#1      _HttpHeaders._addAll (dart:_http/http_headers.dart:74:18)
#2      _HttpHeaders.set (dart:_http/http_headers.dart:89:5)
#3      main (file:///usr/local/google/home/nbosch/projects/dart_repro/foo.dart:6:19)
<asynchronous suspension>
#4      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:301:19)
#5      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

So, if I pass a String to headers.set, should it be encoded for me or is that my responsibility?

[sortie]

Yes that is your responsibility. Dart will decode headers as ISO/IEC 8859-1 per a SHOULD in the standard for interoperability, but it's not recommended to actually send non-ascii data in the headers, so Dart doesn't let you do it in its own requests. We could allow ISO/IEC 8859-1 characters in the request for symmetry, but that probably just hides the fact that it's not UTF-8 for a while longer and leads to surprises and issues with interoperability with systems that don't decode the headers correctly.

If you need to transfer non-ASCII data in the headers, consider using a request body instead, or encoding your data as UTF-8 using URL encoding, base64, UTF-7, or any other scheme; depending on what the remote side supports.

Did that answer your question?

[natebosch]

Thanks! That does answer the question.

Now the question goes back to this package. Our current API doesn't give users a way to accomplish this, and I think it would be breaking to add one. Since it's not something most users should do, we can either say it won't be allowed in this package (and users have to go to dart:io or dart:html directly themselves), or we can try to figure out an upgrade path to get there that is the least disruptive.

cc @zichangg for thoughts.

[zichangg]

Like what @sortie has explained!

I don't think we have a plan to allow users to customize the encoding. I vaguely remembered the conversation with Lasse in one of the sdk issue. It is probably won't be added until next big jump(dart3?).

I think It would be nice to clearly document somewhere. Correct me if I', wrong, I don't think dart:io has clearly written "ISO-8859-1" is used for decoding. Shall we add it in the dart:io? @sortie

For package:http, it might be better to have links to dart:io or dart:html APIs. If users met problems, they will be able to see a detailed explanation. Given the fact that bugs reported here are likely to originate from core libraries, it can save some time for redirecting some issues into dart-sdk.

[sortie]

@zichangg Yes, excellent point, we should document this behavior clearly in dart:io.

[ryanheise]

One use case for setting raw headers without validation is ICY headers. Some servers may serve audio with an HTTP response header like this:

icy-description: ...Title of sound track....

where the title might contain non-ASCII characters if, for example, it is German.

Now the audio player just_audio passes requests through a local proxy to implement some features, and to do this it needs to copy the raw HTTP headers across without modifying them. But because headers.set won't allow this for non-ASCII headers, just_audio cannot proxy some URL requests. In practice, that means a Radio app might not be able to play certain radio stations. I could make the proxy rewrite headers to remove non-ASCII characters, but the consuming app may not be happy that the title was mangled, and that the app wasn't trusted to handle the encoding itself.

It would be nice to have a rawHeaders for this purpose as proposed above.

[hyungtaecf]

This is needed for basically every non-English language... I wanted to use a dictionary API on my application but this limitation just makes it impossible...