No User-Agent header by default

[haikuginger]

It appears that, by default, urllib3 does not send a User-Agent header.

By a plain reading of RFC-7231, it appears that user agents SHOULD generally set such a header unless specifically overridden or suppressed:

A user agent SHOULD send a User-Agent field in each request unless specifically configured not to do so.

I couldn't find prior discussion on this, so it does seem worth talking about. Should we send a User-Agent header by default, and if so, what should it be?

CC @sigmavirus24 in particular for an opinion on the intentions of the RFC. Are we a user agent as described in the document? Or, since we expect to be used for the HTTP-level connectivity of a larger process or library, are we exempt from the requirement?

[sigmavirus24]

So I think it's good for us to set a user-agent. I believe the standard library sets a default as well. That said, I suspect there's a case to be made for not adding a default. Either way, I'm 👍 on adding a python-urllib3/1.21.2 sort of string. When Requests had the extra system information in there, it was rebuked thoroughly because we had too much detail.

[haikuginger]

Okay! python-urllib3/$version sounds good to me, so unless there's some other objection from the team, I'm going to mark this as Contributor Friendly.

[Madhu94]

@haikuginger I will take a shot at this.

[haikuginger]

Thanks, @Madhu94! Let me know if you get stuck, but I believe this should be pretty straightforward.

[user234683]

I'd like to mention that there are use cases where it is undesirable to state the library version being used (since this can vary per user). I'm making a project where anonymity is important, and the fact that urllib sends the python version if no user agent is set caught me off guard a bit, and I had to manually set my own default in the wrapper function I use for network requests in case I forget to set the header when I call it. I don't have a strong opinion on the matter, but I thought I'd mention it.

[ghost]

Would it be alright if I picked this issue up, taking Madhu's patch and the related code review as a starting point?

[sethmlarson]

@DonaCthulhuote It's all yours!

[pquentin]

Closing since #1750 was merged, even if we still need to fix #1750 (comment).