Http authentication

[tpetricek]

It would be nice if the Http methods also allowed specifying credentials (so that we can nicely solve something like this).

(But on the other hand, we do not need to have tooltip that does not fit on a screen :-))

[veikkoeeva]

This is a follow-up from my question in Stackoverflow as to how to use JsonProvider with authorization.

I'm currently on a phone (it can take a few days I'll get properly to check this), but to facilitate things, how shoud one go about adding authorization to JsonProvider? Glancing through the sources I see some Http.fs methods accept headers (AsyncRequest/Request -> InnerRequest mainly) and this looks like being the place to add headers. Would this be a correct assumption to make?

Then it looks like the user name and password parameters should be exposed in JsonProvider.fs for user consumption. Would this be a correct assumption to make?

Assuming these are the main places to grok, I didn't see how to the call chain goes to Http.fs and where the headers come to play.

As a side note, there could be some slight crinkles
http://stackoverflow.com/questions/1702426/httpwebrequest-not-passing-credentials
http://devproj20.blogspot.fi/2008/02/assigning-basic-authorization-http.html (for some more elaborate explanation related to the SO link).

[ovatsus]

I would approach in the following way:

• Also support for using something like http://user:password@url in the url parameter of Http.Request, so XmlProvider, JsonProvider & CsvProvider automatically support that also. Do that parsing somewhere around here: https://github.com/fsharp/FSharp.Data/blob/master/src/Net/Http.fs#L185
• When the sample url contains authentication, disable the generation of the GetSample methods for security reasons (so that the username password are not embedded in the generated code without the user realizing that). See here: https://github.com/fsharp/FSharp.Data/blob/master/src/CommonProviderImplementation/Helpers.fs#L259

[ovatsus]

And don't forget to update the docs here: https://github.com/fsharp/FSharp.Data/tree/master/docs/content/library

[veikkoeeva]

Thanks for the nudge, I wonder why I didn't think of the standard URI scheme in the first place and besides, should other ways be supported?

There's some C# code below, apologies for that, and if I'm rehashing something that's in the code already. I'm currently just documenting my current flow of thoughts here (just in case) and charting the territory, so to speak.

I think the very basic way to get basic authentication supported would be to just add

string credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes(new Uri(uri).UserInfo))
client.Headers[HttpRequestHeader.Authorization] = "Basic " + credentials

I'm fairly positive that at least on Windows one wouldn't need to touch the URI (say, to remove credentials or anything) in any way and concerning my small code snippet, only a string.IsNullOrEmpty check should be added, if even that.

But it looks there could be subtle issues left lurking going down this route. Two of which are that authorization with negotiated authentication or with proxies wouldn't work. So, an improved change could be something like

var cc = new CredentialCache();
cc.Add(new Uri(uri), "BASIC", new NetworkCredential("username", "password"));
cc.Add(new Uri(uri), "NEGOTIATE", CredentialCache.DefaultNetworkCredentials);
var client = new WebClient();
client.Credentials = cc;
if(client.Proxy != null) client.Proxy.Credentials = cc;
//client.PreAuthenticate = true;

Where username and password are obtained from Uri.UserInfo by splitting it on colon (other code and creating credentials of course with proper checks).

I'm not sure how I could write tests for the authorization code, but for what I understand from the documentation CredentialCache would be in charge of choosing the correct authorization credentials and would take care, for instance, they aren't accidentaly exposed in presence of redirects.

I used the following resources to gleanse more information on how WebClient authorization is supposed to work

• Network Class Library Team (System.Net): Custom HTTP Authentication Schemes
• Network Class Library Team (System.Net): System.Uri FAQ
• Rick Strahl's Web Log: .NET WebRequest.PreAuthenticate – not quite what it sounds like
• Buck Hodges: Authentication in web services with HttpWebRequest

[ovatsus]

I though of a using the uri just for helping passing the parameters, but it's probably safer to remove that from the uri before constructing it, and handling separately.

Ideally, instead of that, we'd add specific parameters for Http.Request instead, but that wouldn't work at compile time when getting the schema for JsonProvider or XmlProvider, as you only have the url, unless you add user and password parameters to all type providers, which will be a bit of parameter creep, they already have a lot of them.

Testing will be a tricky bit, we currently have zero testing of Http.Request except for it's basic usage throught passing url's in the sample parameter of the type providers, but it would be nice to improve on that

Do what you think it's best and then we'll review and iterate

[ovatsus]

Hi @veikkoeeva, are you getting along with this, or do you need help with something?

[veikkoeeva]

Hi!

I've made progress and I have code that allows amongst other things one to authenticate via HTTP Basic. It has just been a bit bumpy at work, so I haven't had the time to clean things properly and write rationale for the actions I've taken. I'm planning to do something this evening (kids and wife allowing :)), and I expect to submit pull requests no later than during the coming weekend.

Currently in my code there is HTTP Basic, Digest and NTLM/Kerberos (negotiate), of which I'm able to test HTTP Basic (I need to export the code to a domain joined work machine to see if negotiate can be get to work that way). Basically it's done as you can see in the previous snippet on using CredentialCache -- the username and password aren't added to negotiation based authorization currently. I also scrub the URL from the credentials (I haven't gone yet to disable samples). In case of HTTP Basic, though, the credentials are passed as plaintext already (as mandaged by "some" RFC, so scrubbing them from the URL wouldn't provide more security, I believe. Still, perhaps it's good hygiene to remove the credentials from the URI. For security reasons it doesn't seem reasonable to take the credentials in the URI and use them to build negotiation credentials.

Then I have done some research (on more authoritative sources than previous links) as how do (Windows) proxies work and perhaps what kind of authorizations scenarios could be supported and the problems people encounter with authorization. For instance, there is a "double hop issue" when accessing SharePoint, which could be solved by adding two sets of credentials. This particular problem is described succintly at Pass credentials along to new WebRequest, as well as solutions (also in comments). Then there are various issues with proxies, something what looks like being a bug in Digest authorization and so forth, but I'll see if I can write a more coherent memo about things I've come across.

Oh, I believe one could also leverage Windows Credential Manager (or major browser credential sotres), albeit they look like being rather insecure. I suspect I just mention these in the memo and drop them for further consideration at this point.

[veikkoeeva]

This feature lacks documentation, but I'll try to update it today (it's about 17 o'clock here) or the latest tomorrow. I seem to lack time currently, so I pushed it here so we can iterate as discussed.

One place to test the code is at http://eu.httpbin.org/ . Upon reading the code written before me I suspect I semi-accidentally recreated some of the functionality already present in Helpers.fs in some form, mainly regarding on how to recognize if the provided "uri" is a valid one (especially a valid URL).

Perusing some discussion I noticed I probably could hook myself (I'm on Windows 8.1) to the build engine before committing and syncing, but that'd be taken some time to do. Hmm, what else, anything to improve my F# or how to better use GitHub or "behave" in FSharp.Data is fair game too.

[ovatsus]

Yes, we already have the docs using httpbin, I think that's a good option. But these things are always hard to test, so If you only build and test manually it's fine.
I'll wait for you to update the pr with some docs, as I'm busy with other stuff right now, and then I'll look into it and merge in the next couple of days.
Thanks for your work!

[veikkoeeva]

I added some documentation.

You are welcome, and thank you for lending a hand and having patience. It looks like you'll need it, as Travis says the build fails with a message regarding SecureString. I have no clue how to fix that (other than using plain strings, which might be just as secure in this case) as the code compiles just fine on my box. Additionally GitHub says I want to merge master in my pull request, but Travis wants to me belive my changes are already in the master.

In any event, I'll jot down some notes here regarding authorization.

[ovatsus]

Maybe SecureString is not available on mono? I'll have a look later

[veikkoeeva]

It looks like being there in Mono. It looks like Mono's version of NetworkCredentials doesn't support SecureString. For comparison, here the version of MS flavor of .NET. I need to run now, but I'll file a bug later in the evening/niught (unless someone does it before me). Who knows, maybe I will eventually even add the overloads. In the meantime, how should we proceed?
<edit 2014-02-24: It looks like the Mono version lacks only the constructor overloads, but provides a property setter for SecureString password. So, perhaps the course of action is to set the credentials via properties.

Oh, I can iterate this more in case it's not a problem my commits are somewhat spotty and concentrate to weekends. Maybe it's clear, but I also don't mind if someone makes the code better or rearranges the various parts of code I added.

To add more context on what I did and why: Currently I just added support for parsing the userinfo part of the URL, added those credentials to the WebRequest the library will use and removed the credentials from the URL used. I think this scenario needs to be supported since it's a part of the standard URI scheme. However, whilst pondering – unfortunately rather Windows centrically, please, bear with me – what scenarios should and could be supported and what kind of troubles people encounter, I learned more on the authentication story and I believe there is enough of a case to consider exposing a user supplied CredentialCache parameter and/or perhaps some factory methods to build them. Regarding this particular ticket, one could just write

let credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes((new Uri(uri)).UserInfo))
client.Headers[HttpRequestHeader.Authorization] <- "Basic " + credentials

and get the HTTP basic authentication to work. However, if redirects were allowed, the same credentials would be sent to the new address. In that case

• It may not be desired as the redirected address may not be deemed trustworthy.
• The resource may not accept those particular credentials, but some other ones.

By using the CredentialCache one gains control on how the credentials are used. There's a succinct summary on this at Network Class Library Team (System.Net): Custom HTTP Authentication Schemes

WebRequest by default facilitates automatic request redirection from site to site. However if a
server you have logged into decides to redirect you elsewhere, WebRequest will remove its Credentials property before automatically redirecting to prevent your username and password from being exposed. If you are aware that redirects will take place, you can use the CredentialCache class to manage what servers are allowed access to your credentials. A CredentialCache pairs credentials with the URIs and authentication schemes they’re allowed to be used with.

(As a side note, the just mentioned blog writing has a code example on how to create credentials for Google logon, where a token needs to be retrieved first. Maybe those ideas could be used to unify the interface (thinking of Twitter etc.) if one ever decides to support passing CredentialCache as a parameter.)

Adding a set of credentials to CredentialCache and passing it to the WebRequest looks like is also the correct way to handle proxies that require authentication, redirects and what comes to possible mixed authentication schemes. I suspect this is so, I’m not entirely sure. By default WebRequest looks like is using the same proxy settings as IE, which is probably what is generally preferred. The settings can also be altered with the proxy element in .config files. As for proxies in Windows in general at least two authoritative looking posts on the matter look like being

• IEInternals: Proxy-Authentication breaks many applications
• IEInternals: Web Proxy Configuration and IE11 Changes
  A perhaps worthy point is that there’s a user voice ticket on making proxies work better with .NET at Fix it so that .NET apps can access HTTP thru authenticating proxy server by default.

Further ideas (probably bad ones, but I just slap them here too), and why I also brought up the "Google credentials" are elaborated from a post by Jeff Sanders on Using HttpWebRequest with Credential Manager. That is, the user is prompted to enter credentials and they can be saved to the system. This idea could be elaborated on using browser stored credentials, as shown in How Browsers Store Your Passwords (and Why You Shouldn't Let Them) (or rubbing Credential Vault still further Saving Credentials on Windows Computers). These latter points and the Google case mentioned in parenthesis are something I think could be considered as factory methods.

[veikkoeeva]

It looks like I created a new issue, namely #451, whilst messing around in Github. Sorry about that. Anyhow, I compile here some more notes and issues if and when integrated authentication should be added. Maybe these are of use to someone else too.

• Feroze Daud's WebLog on MSDN: HttpWebRequest and PreAuthentication
  A nice primer on what happens behind the scenes.
• Feroze's newer blog has two interesting posts on networking on Windows: How to share NTLM connection across HttpWebRequest objects and A History of Microsoft's HTTP Client Stacks.
• One particular gotcha Ayende@Rahien mentions at Mixing Integrated Authentication and Anonymous Authentication with PreAuthenticated = true doesn’t work.
• And a weid scenario that seem to involve PreAuthenticate: A POST or PUT request may fail when you use the HttpWebRequest class to send lots of data on a computer that is running the .NET Framework.
• RestSharp has added support for DefaultCredentials too, and they seem to have discovered something Silverlight specific: Added support for UseDefaultCredentials #455, namely RegisterPrefix (also here and here for more examples).

I should have some time during the next weekend to work on this, but of course others can too.

[veikkoeeva]

I filed a bug report to Xamarin Bugzilla regarding the missing NetworkCredential constructor SecureString definitions.

[ovatsus]

Cool, in the meantime, can we not use securestring to make it work on mono?

[liammclennan]

Does this support using the default credentials of the caller, ala http://msdn.microsoft.com/en-us/library/system.net.credentialcache.defaultcredentials%28VS.80%29.aspx ?

[veikkoeeva]

@ovatsus Both the MS and Mono implementation has a property setter that accepts the password as SecureString, I changed the constructor calls to use object initializers (syntax) and we still have SecureString. The constructor overlaods were added to MS .NET in version 4.0.

@liammclennan Yes, but I believe one should use DefaultNetworkCredentials instead, which was added in .NET 2.0 (see also Buck Hodges' blog post). Worthy of a note is also that in enterprise setting, where NTLM/Kerberos (negotiation) is widely used, there often seem to be redirects and maybe even authenticating proxies (that is, a mixture of, say, NTLM and HTTP Basic Auth). Good keywords to search for potential usage scenarios could be something like "double-hop SharePoint", which probably apply a lot of other installations too, like TFS. If one thinks of it, CMSs is probably one of the key usage scenarios of type providers in enterprises in addition to databases.

The redirects is one of the reasons I used CredentialCache instead of writing directly the HTTP Auth headers to the request. But don't take just my writing for it, I kind of rushed this during some nights and I'm not intimiately familiar this stuff besides "I have done something, it has been messy and somehow I got it working case-by-case" and that's why I provide here some of the links I perused during my background studies.

Also, it could be that CredentialChage and AuthenticationManager could be made to implement Google and Twitter like authentication schemes "cleanly", but maybe that is something for future refactoring if that looks like a necessary thing to do. For what it's worth, maybe we should consider exposing CredentialCache or some other mechanism to type provider users in case finer grainer control is required.

[veikkoeeva]

A note for myself (and @liammclennan) that the default Basic Auth module does not work as expected (correctly, one might say) depending on the user system locale and credentials. Source: Replacing the built-in Basic Authentication Module to support non-English characters in a HttpWebRequest . Briefly, by default the credentials are encoded using the default code page. To quote

But wait, let us say I typed the user name in German. If my local OS language is say German and my server OS language is English, the local translation of German chars will succeed, but when the credentials will arrive at the remote server they will fail to translate back.

I haven't verified this myself, but feels plausible and I should take care of right from the get-go.

[veikkoeeva]

A note for the interested: I implemented Basic authorization as IAuthenticationModule which encodes the authorization in UTF-8 and it works as far as I can see testing it manually. I suspect the same treatment should be given to Digest, but I'll post-pone that (maybe indefinitely) so that I get to see OAuth too, but I'd rather not to promise too much about that. :) This means we still have basic auth that works, now even more, I think. I'll clean-up the implementation and commit to GitHub probably today (after I've put kids to sleep).

Also, this isn't actually as simple as it looks like being if one wants to support clients and servers widely. There's a question of this in SO What encoding should I use for HTTP Basic Authentication?, wherein it's concluded shortly the spec is unclear and that "ISO-8859-1" or "undefined" are the encodings used. Also to quote

Also relevant, no current browser does this. They use utf-8 (Chrome, Opera), iso-8859-1 (Safari), the system code page (IE) or something else (like only the most significant bit from utf-8 in the case of Firefox).

There's a new version of the specification in the makings, available (navigating from the aforementioned SO link) at An Encoding Parameter for HTTP Basic Authentication (main here):

A. Deployment Considerations
A.1 User Agents

User agents not implementing this specification should continue to work as before, ignoring the new parameter.

User agents which already default to the UTF-8 encoding implement this specification by definition. Note that some user agents also have different defaults depending on whether the request originates from page navigation as opposed to a script-driven request using XMLHttpRequest [XHR].

Other user agents can keep their default behavior, and switch to UTF-8 when seeing the new parameter.

A.2 Origin Servers

Origin servers that do not support non-ASCII characters in credentials do not require any changes.

Origin servers that need to support non-ASCII characters, but can't use the UTF-8 encoding will not be affected; they will continue to function as well or as badly as before.

Finally, origin servers that need to support non-ASCII characters and can use the UTF-8 encoding can opt in as described above. In the worst case, they'll continue to see either broken credentials or no credentials at all (depending on how legacy clients handle characters they can not encode).

[veikkoeeva]

Hmm, it shows a branch that doesn't exist. In any event, the latest changes are at https://github.com/veikkoeeva/FSharp.Data/tree/158-Http-Authentication.

I can send a pull request to that one, or do something if someone wants to have changes. For instance, integrated authentication isn't done at the moment. I could put it on for all requests by default, but it may be better to expose the credentials to the type provider users. Maybe in some fashion like Relentless/Https.fs is doing it:

|> withBasicAuthentication "myUsername" "myPassword"

Wherein there could be other authentication schemes too, like OAuth2. Using the standard URI scheme, which there is at the moment, doesn't look like getting us that far. I feel I'm not the right guy to think about that.

There's a comment in the code where Basic Auth and Digest is added that

Doing this like here isn't ideal. Instead, it seems like an approach where the user/programmer defines which credentials should be added to the cache should be preferred. It could be in that case the type provider interface should expose a method to add directly credentials either implementing ICredential or inheriting from NetworkCredential.

[veikkoeeva]

I just found out the syntax to customise WebRequest calls via The Http of FSharp.Data and proxy server (as an aside, this can be customised also in .config settings). So, this could be used to leverage CredentialCache too, also if implementing OAuth.

A rather complete looking treatment of authenticating proxies is written by Jeff Sanders in How to accommodate Authenticating Proxies in .NET (except that one may need to add the credentials also to the request itself in case I interpretet "transparent proxy" here correctly).

[ovatsus]

I've been looking at this but I haven't merged it yet because I was trying to find a way to at least make part of it work in the portable versions. I'd prefer to have a smaller subset of supported features common to all platforms, and add the rest as samples in the docs. I hope to have time to look into it with more detail soon

[veikkoeeva]

How could I find out which platform is being used? Something in AssemblyResolver.fsx perhaps? I can take a quick stab at it tomorrow if I find out how to check which platform the code is being run on.

Otherwise I'm afraid I'm not familiar with compatibility libraries. From the looks of it, CredentialCache and `PreAuthenticate``aren't available. However, with a quick search I found a SO post wherein someone digs them out by reflection: Replacement for PreAuthenticate in portable class libraries. If this works, it'd be good, but I haven't checked this as I'm about to go to sleep.

Then also Silverlight 4 – Credentials, we’ve got it! (note RegisterPrefix). Also straight adding the headers seem to work by looking at code in the Internet. I'm not sure how to handle proxies and redirects, but I managed to dig up some examples to do OAuth on portable libraries courtesy of LINQ to Twitter.

Regarding OAuth, I asked if a friend of mine could lend a hand with details, as he's implemented single-sign-on/out system featuring also OAuth just recently (and is in process of writing a blog series of some aspects of it).

[ovatsus]

All platforms are on the solution now, so you just need to build. I have it merged with the latest changes, I'll push it to a branch so you can carry on from there.

[ovatsus]

It's here: https://github.com/fsharp/FSharp.Data/tree/HttpAuthentication
I suggest you clone that branch and carry on from there
You'll see that the Portable7 and Portable47 projects are not compiling

[simra]

I'm interested to learn the status of this issue. Looks like no updates for exactly a year and the branch was deleted on the other thread. (#630 above). I'm interested to load a type provider and fetch data using windows integrated auth.

[cboudereau]

Hello you can load first the data with a webrequest or other and then use
the load method of the xml type provider.
On Mar 23, 2015 9:10 PM, "simra" notifications@github.com wrote:

I'm interested to learn the status of this issue. Looks like no updates
for exactly a year and the branch was deleted on the other thread. (#630
#630 above). I'm interested
to load a type provider and fetch data using windows integrated auth.

—
Reply to this email directly or view it on GitHub
#158 (comment).

[cboudereau]

There is another option : there is a function with a webrequest parameter.
You can set the webrequest before the xml type provider use it.
On Mar 23, 2015 9:14 PM, "Clément BOUDEREAU" cboudereau@gmail.com wrote:

Hello you can load first the data with a webrequest or other and then use
the load method of the xml type provider.
On Mar 23, 2015 9:10 PM, "simra" notifications@github.com wrote:

I'm interested to learn the status of this issue. Looks like no updates
for exactly a year and the branch was deleted on the other thread. (#630
#630 above). I'm interested
to load a type provider and fetch data using windows integrated auth.

—
Reply to this email directly or view it on GitHub
#158 (comment).

[dsyme]

Just to mention that I was using the GitHub API today, unauthenticated, and quickly hit the rate limitations of that API. It would have been great if I could have very seamlessly switched to OAuth or Basic authentication - including for both design-time and compile-time web requests

It feels like a really important feature to add systematically to FSharp.Data, and it would make even much more powerful.

See also #453 (comment)