[SR-2429] Regression: HTTPURLResponse allHeaderFields is now case-sensitive

[swift-ci]

Previous ID	SR-2429
Radar	rdar://27925651
Original Reporter	alexsporn (JIRA User)
Type	Bug
Status	Resolved
Resolution	Done

Environment

Xcode Version 8.0 beta 6 (8S201h)

Additional Detail from JIRA

Votes	27
Component/s	Foundation
Labels	Bug
Assignee	@airspeedswift
Priority	Medium

md5: 2495a8cbf7aef0a067f006a363dbbfce

Issue Description:

HTTPURLResponse normalizes header names following RFC 2616 as described in https://developer.apple.com/reference/foundation/httpurlresponse/1417930-allheaderfields

Access to these header values is case-insensitive as described in the documentation. This is broken in the latest Xcode8 beta6. The access is now case-sensitive.

Additionally the "ETag" header name is normalized to "Etag" instead of "ETag" as described in RFC 2616

This works correctly on Xcode8 beta4

Steps to Reproduce:
Run following Swift code:

let url = URL(string: "http://www.apple.com")!

let headers = [
    "ETag" : "123456",
    "content-type": "application/json",
    ]

let urlResponse = HTTPURLResponse(url: url,
                                  statusCode: 200,
                                  httpVersion: nil,
                                  headerFields: headers)

print("ETag: \(urlResponse?.allHeaderFields["ETag"])")
print("Etag: \(urlResponse?.allHeaderFields["Etag"])")
print("ETAG: \(urlResponse?.allHeaderFields["ETAG"])")
print("etag: \(urlResponse?.allHeaderFields["etag"])")
print("content-type: \(urlResponse?.allHeaderFields["content-type"])")
print("Content-Type: \(urlResponse?.allHeaderFields["Content-Type"])")
print("CONTENT-TYPE: \(urlResponse?.allHeaderFields["CONTENT-TYPE"])")

Same code in Objective-C works as expected:

NSURL *url = [[NSURL alloc] initWithString:@"http://www.apple.com"];

NSDictionary *headers = @{
                          @"ETag": @"123456",
                          @"content-type": @"application/json",

                          };

NSURL *url = [[NSURL alloc] initWithString:@"http://www.apple.com"];

NSDictionary *headers = @{
                          @"ETag": @"123456",
                          @"content-type": @"application/json",

                          };

NSHTTPURLResponse *response = [[NSHTTPURLResponse alloc] initWithURL:url
                                                          statusCode:200
                                                         HTTPVersion:nil
                                                        headerFields:headers];

NSLog(@"Headers: %@", response.allHeaderFields);

NSLog(@"ETag: %@", response.allHeaderFields[@"ETag"]);
NSLog(@"Etag: %@", response.allHeaderFields[@"Etag"]);
NSLog(@"etag: %@", response.allHeaderFields[@"etag"]);
NSLog(@"ETAG: %@", response.allHeaderFields[@"ETAG"]);
NSLog(@"content-type: %@", response.allHeaderFields[@"content-type"]);
NSLog(@"Content-Type: %@", response.allHeaderFields[@"Content-Type"]);
NSLog(@"CONTENT-TYPE: %@", response.allHeaderFields[@"CONTENT-TYPE"]);

Expected Results:

Headers: Optional([AnyHashable("Content-Type"): application/json, AnyHashable("Etag"): 123456])
ETag: Optional(123456)
Etag: Optional(123456)
ETAG: Optional(123456)
etag: Optional(123456)
content-type: Optional(application/json)
Content-Type: Optional(application/json)
CONTENT-TYPE: Optional(application/json)

Actual Results:

Headers: Optional([AnyHashable("Content-Type"): application/json, AnyHashable("Etag"): 123456])
ETag: nil
Etag: Optional(123456)
ETAG: nil
etag: nil
content-type: nil
Content-Type: Optional(application/json)
CONTENT-TYPE: nil

Version:
Version 8.0 beta 6 (8S201h)

[swift-ci]

Comment by Alexander Sporn (JIRA)

This is still broken with Xcode 8.0 GM (8A218a)

[swift-ci]

Comment by Aaron Brager (JIRA)

I can reproduce this. Additionally, it is coming up on Stack Overflow (for example https://stackoverflow.com/questions/40152483/httpurlresponse-allheaderfields-swift-3-capitalisation/40152676#40152676)

[swift-ci]

Comment by Aaron Brager (JIRA)

alexsporn (JIRA User) I would suggest updating the priority from Medium since this is a violation of the HTTP spec and is a lot of effort to work around.

[swift-ci]

Comment by Alexander Sporn (JIRA)

aaron (JIRA User) I think the Priority is reserved for project leads. I am not allowed to change it.

[mdiep]

It's not just that the headers aren't canonicalized. Lookups are also case-sensitive when the documentation clearly says that they are case-insensitive:

The returned dictionary of headers is configured to be case-preserving during the set operation (unless the key already exists with a different case), and case-insensitive when looking up keys.
(https://developer.apple.com/documentation/foundation/httpurlresponse/1417930-allheaderfields)

[swift-ci]

Comment by Glen Huang (JIRA)

Just got bitten by this one too. Xcode 9.0 GM 9A235

[dmcgloin]

Uggh. This seems like a serious bug. Documentation is wrong. Implementation is broken. Vote for priority increase. Xcode 9.1.

[swift-ci]

Comment by Ryan Romanchuk (JIRA)

woof. This is going to be a very bad bug and break a lot of functionality for anyone who just released with 9. Any app that uses response headers for pagination is going to have a bad time. It's extremely subtle regression that's probably not going to get caught by even the best test coverage.

[swift-ci]

Comment by Charles Joseph (JIRA)

I just got bitten by this issue too — took a bit to figure out why some URL responses weren’t being handled properly (Xcode 9.1, Swift 4.0.2)

[swift-ci]

Comment by Shawn Erickson (JIRA)

Yeah we just ran across this ourselves... luckily our logic had sufficient fallbacks in place to deal with it well enough for users that happen to go via a proxy that tweaks the case of header keys. Version 9.2 (9C40b) / Swift 4.x

I filed rdar://38812951

[swift-ci]

Comment by Paul Bruneau (JIRA)

How is it even remotely possible that this isn't fixed yet?

[swift-ci]

Comment by Sander de Koning (JIRA)

This needs to be fixed asap. This is a serious bug that needs attention from the team!

[swift-ci]

Comment by Javier Ortiz (JIRA)

It would be nice fix a crucial bug reported 2 years ago.

Could you change the documentation if you're not going to fix it?

[swift-ci]

Comment by Tyler Hunnefeld (JIRA)

So many teams have been forced into inefficient solutions to address this. @airspeedswift Is there an update on the status of the bug? If this is not deemed to be an issue worth fixing, could the (incorrect) documentation specifying that this is a case insensitive lookup be updated to avoid tripping up developers in the future?

https://developer.apple.com/documentation/foundation/httpurlresponse/1417930-allheaderfields)

[guoye-zhang]

Workaround: (response.allHeaderFields as NSDictionary)["etag"]

[swift-ci]

Comment by Paul Bruneau (JIRA)

Pardon the question, but how is that a workaround? NSDictionary is case sensitive for text keys, so this won't help.

Apple's objc implementation of `allHeaderFields` seemed to use some private case-insensitive key-value data structure that their Swift implementation does not have.

Or are you saying this in fact works?

[guoye-zhang]

Yes, this works. Under the hood, (response.allHeaderFields as NSDictionary)["etag"] disables bridging and calls the Objective-C implementation of the private NSDictionary subclass in CFNetwork directly

[swift-ci]

Comment by Christian Menschel (JIRA)

Please use "public func value(forHTTPHeaderField field: String) -> String?" which is case insensitive.

instead of "allHTTPHeaderFields["X-Something"]" which would be case sensitive

[dmcgloin]

tapwork (JIRA User) AFAIK, that function is available on `NSURLRequest`, not `NSHTTPURLResponse`.

[swift-ci]

Comment by Steven Kramer (JIRA)

Countless hours of wasted developer effort later…

The documented behavior seems very sensible, and so does tapwork (JIRA User)'s proposal, if it were available. Fixing the documentation would be a least-possible-effort but still welcome solution.

[swift-ci]

Comment by Pablo Cobucci Leite (JIRA)

Just got hit by this issue too, Xcode 10 - Swift 5. Voting for this issue.

+1 on at least fixing the docs, but the definitive solution should be fixing the API as headers should be case-insensitive.

@guoye-zhang's suggested workaround works fine (response.allHeaderFields as NSDictionary).

[swift-ci]

Comment by Kevin Cassidy Jr (JIRA)

Update: HTTPURLResponse now includes func value(forHTTPHeaderField field: String) -> String?

https://developer.apple.com/documentation/foundation/httpurlresponse/3240613-value

For iOS/tvOS 13, macOS 10.15, and watchOS 6 the example could be written as urlResponse.value(forHTTPHeaderField: "etag")

One could also implement this as an extension for older OS versions using the NSDictionary cast workaround above.

[beccadax]

Hi everyone,

There's no way for us to keep HTTPURLResponse.allHeaderFields from being bridged to Swift as a case-sensitive dictionary, so this bug could only be fixed indirectly.

To provide a way to do a case-insensitive lookup from Swift, Foundation has added a HTTPURLResponse.value(forHTTPHeaderField:) method. The Swift documentation for HTTPURLResponse now recommends using this method instead of allHeaderFields to look up headers by name.

This new method is available in the OSes released with Swift 5.1 (macOS 10.15/iOS and tvOS 13.0/watchOS 6.0). To backwards deploy on older OSes, using (response.allHeaderFields as NSDictionary)[someHeaderName] is a supported workaround that is expected to continue working in future OSes. The method was not included in Swift 5.1 or 5.2's Corelibs Foundation, but as of SR-12300, it's now in nightly snapshots and will probably be in Swift 5.3's Corelibs Foundation.

This bug has therefore been fixed by providing an alternate Swift-compatible API for the task. In the future, please use value(forHTTPHeaderField:) or, if necessary, the as NSDictionary workaround to look up headers case-insensitively.

Thanks!