Case insensitive headers

[Cyclic]

Using Grape 1.2.3 and started experiencing this issue after upgrading. Ruby 4.6.2p47, Rails 4.2.11.1 I've found that headers get lowercased, except the first character 'X', when using RSpec tests so my lookups for the header key fails. For example, when a request is made through my app, the case is converted to "X-Auth-Token", but when access through RSpec the same header appears as "X-auth-token".

I understand that headers are case insensitive, but what is responsible for converting the header case in grape? Ideally, the case is either all caps, all lowercase, or camel case - or there is a method provided in grape to lookup header keys with case insensitive matching. I don't want to write a wrapper as stated here: https://www.ruby-forum.com/t/case-insensitive-hash/51767/2

I'm at a loss on how to best access headers so that they are case insensitive, so any help is appreciated.

Thanks in advance.

[dblock]

Grape does not convert headers lowercase/uppercase AFAIK, there's certainly no such code. I don't believe you/we should be doing it. Did you bring in another dependency that's doing this? What did you upgrade from and to and can you narrow it down to grape?

[Cyclic]

Thank you for the prompt response, and I very much appreciate your insight. I suspect that it's either faraday, httparty, or net::http; although, I have not been able to track it down. I thought others might have experienced this and have a general solution that might prevent me from having to write a wrapper class, and I agree, this is not something we should have to handle. I'll need to see if any libraries that I've added are doing this, but it only appears when upgrading from grape <= 13 to >= 15 (I suspect), but maybe even 14. I'll need to try out other grape versions to find out (like testing 14 with other fixes that I had to put in place). It definitely only happens when I bump the grape version.

EDIT: Turns out to be the ! in capitalize for some versions of Ruby (created a PR, but unsure if removing ! covers all cases):

def build_headers
      headers = {}
      env.each_pair do |k, v|
        next unless k.to_s.start_with? HTTP_PREFIX

        k = k[5..-1].split('_').each(&:capitalize!).join('-')
        headers[k] = v
     end
     headers
end

[dblock]

Coming back from #1881, the headers helper was added in f6f585e and from day 1 it converted the headers case into pascal-case. That is by design and I was wrong about whatever I said above.

Let's find out where the headers such as X-Auth-Token become X-auth-token, because I would expect Grape to not do this (and convert them to X-Auth-Token). Do you think you can cook up a simple repro @Cyclic?

[dblock]

I opened #1882 to document what we do to headers.

[Cyclic]

I was able to reduce the project down to a minimal number of files to reproduce the header casing issue.

Please find the repo here:
https://github.com/Cyclic/rocket.api

Just clone the repo, run bundle install; rake db:create;

Execute rspec on spec/api/v1/core/reproduce_spec.rb

You should notice that X-NL-Auth-Token is all caps going into the get request, and then gets lowercased if you breakpoint in user_helpers.rb line 4, after the call to the API is made.

Just as a reminder, removing the ! from k = k[5..-1].split('_').each(&:capitalize!).join('-') seems to work in some cases, but I have yet to figure out why it doesn't work in others.

Thanks in advance and drop me a note if you have any trouble reproducing.

[dblock]

Having hard time running the spec. It complains about missing GuestUser for one.

A simple thought in the meantime, X-NL-Auth-Token is not the capitalization of the code above, it should be X-Nl-Auth-Token, with a lowercase l. Does that fix the problem?

[Cyclic]

Not sure why, but now I'm getting missing endpoint. I commented out the GuestUser code. If you can get the endpoint loaded, you will see that the header becomes X-nl-auth-token ...

I updated the repo with the changes to see if it works. I'm still trying to get it to work (it was working last night), but after running bundle update this morning, the spec stopped working. Going to try to get the endpoint working again so you can repro, but see if you might be able to figure out why it says endpoint is missing - it's mounted.

Thanks!

[Cyclic]

Very odd! I was able to set a breakpoint on line 4 of user_helpers, after running bundle update, the headers are the correct case. X-NL-Auth-Token is now in the headers. Going to keep trying to repro.

Update:
Disregard that last message above. I forgot that I removed ! in my grape response.rb ... So, even though the error for missing endpoint is still there, you can see the case issue with 'puts headers' in user_helpers.rb line 4.

[dblock]

@Cyclic Care to get this into a state where I can see an actual failing spec? The current spec fails with "no such endpoint" if I look at last_response.body. This isn't what you expect I imagine.

[dblock]

I got this. The issue is that the header is coming into the Rack env object as HTTP_X-NL-Auth-Token and not HTTP_X_NL_AUTH_TOKEN. The third parameter in Rack::Test::Methods is a Rack env object, not a set of headers. This is well explained here, too.

So you either need to send a header as in a Rack object, ie.

  def authenticated_headers(user_auth_token)
    {'HTTP_USER_AGENT' => user_auth_token.name, 'HTTP_X_NL_AUTH_TOKEN' => user_auth_token.auth_token}
  end

and

get '/v2/core/reproduce', {}, authenticated_headers(user_auth_token)

Or as a header

header 'X-Nl-Auth-Token', 'token'
get '/v2/core/reproduce'

I'll close this, but feel free to tack on more questions!

[dblock]

@Cyclic Now that the mystery is solved, would you maybe give #1882 a try?

[Cyclic]

@Cyclic Now that the mystery is solved, would you maybe give #1882 a try?

Yes, thank you for the insights! I appreciate your patience and helping me resolve the issue. That article explains it perfectly, so I'll cite that and make it clear in the documentation. I had questions about why it was using the env, but could not find any specific mention of why env was used in the build_headers method. Thanks again!

[Cyclic]

Any idea why there is a missing endpoint error? The spec was working when I first created it.

[Cyclic]

Ok, new issue. After changing all of the headers to HTTP_X-Nl-Auth-Token, checking on the other end after the call, I get Http-X-Nl-Auth-Token ... shouldn't the HTTP_ be removed as before? Sorry for the questions, but this will all go towards the documentation and want to make sure I understand what type of behavior to expect. I updated the repo again.

[dblock]

HTTP_ is a prefix added into the Rack env to signify that those are HTTP headers. So either you specify the header as X-Nl-Auth-Token or put it into env as HTTP_X_NL_AUTH_TOKEN.

I can look at the endpoint error later, but my guess is that you're not mounting the API into Rails routes properly or under a different path.

[dblock]

I wrote this up in https://code.dblock.org/2019/07/06/handling-of-http-headers-in-grape-rack-rails.html and added examples here for Rack and here for Rails.