Bring the async stuff from PowerPack into FSharpx.Core

[ovatsus]

https://github.com/fsharp/powerpack/blob/master/src/FSharp.PowerPack/Async*

I reference PowerPack in most of my projects just to get the AsyncReadToEnd from StreamExtensions in the PowerPack.
It would make sense to add that to FSharpx.Core

[forki]

Did you see https://github.com/fsharp/fsharpx/blob/master/src/FSharpx.Core/CSharpCompat.fs#L523 ?
Does it help?

[ovatsus]

Humm, haven't noticed that FSharpx depended on FSharp.PowerPack.dll, as it isn't included in the NuGet package. That means that the NuGet package is broken, if we use FSharpAsyncEx, we also have to get the PowerPack NuGet package, otherwise we'll get compilation errors.
Anyway, wouldn't it be better to move both that and all the stuff FSharpx is using from Powerpack inside FSharpx and remove the dependency altogether?

[forki]

+1

.... waiting for pull requests ....

[ovatsus]

Cool, I'll send one soon

[mausch]

Is PowerPack being officially deprecated / dropped support from Microsoft? +1 for including PowerPack's async and collections (I think this was in a separate issue?) if that's the case...

[ghost]

Hi @mausch,

Are you on the F# open source google group where this has been discussed recently? We're working out the best way to take the pieces forward. The PP will still be there and won't be changing. But I'm OK with things being replicated into newer project if they are a good fit and/or redesigned.

I see the current remit for FSharpx as:

• commmunity provided extensions to existing FSharp.Core functionality Async, Seq, Event, Observable, Quotations etc.
• STM
• functional programming collections (in a separate package)
• type providers (in separate packages)

This means the corresponding elements in the PP can and should be integrated.

Some changes I recommend to FSharpx.Core are:

• The FSharpx.DataStructures functional collections should go in a separate package and namespace called FSharpx.Collections.Functional - they are great but independent.
• The "Prelude" operators should not be auto-opened - indeed I think there should be no "Prelude" at all. It gives the impression you are trying to change some fundamental design choices in F#. For example, F# has no flip/curry/uncurry operators for good reasons. Code that uses these is often unreadable and unmaintainable and using them is just not good F# coding style. They give the impression that FSharpx is trying to turn F# into Haskell which is fine but it should be in a package FSharpx.Compatibility.Haskell.
• There are some odd naming choices - e.g. it seems module 'Strings' should be renamed 'String' so the functions merged with the String.* operations in FSharp.Core.
• You have too many things in the 'FSharpx' namespace. Programming abstractions like 'Lens', 'Iteratee' and 'TreeZipper' needs to be qualified and perhaps be in a different package.such as FSharpx.Collections.Functional.
• You really need some auto-generated documentation :)

Please do not integrate the following form the power pack:

• math components, as has been discused elsewhere, they belong in Math.NET F#'s support if anywhere
• FsLex/FsYacc. These are stable and useful parsing technologies, but one of the things that made the PP very difficult to take forward was the inclusion of these in the installer for the PP, since they are really tools that require MSBuild tasks, tool support etc. I do not suggest you get into parsing technologies in FSharpx, at least not yet. Set some boundaries. If they were refactored as type providers it may be more reasonable.
• the OCaml compatibility stuff. This should only come into FSharpx if there is a really concerted belief that FSharpx should contain compatibility bindings (e.g. FSharpx.Compatibility.Haskell)
• the documentation generator unless you are using it yourselves
• the CodeDom generator (this is not in good enough shape to integrate - a modified copy now seems to be in use in the MonoDevelop F# bindings anyway)

Thanks
don

[mausch]

Thanks Don, I'm subscribed to the google group but I seem to have missed that discussion.

Also thanks for the advice, I agree with most of your concerns except for the comments about the prelude. We're certainly not trying to turn F# into Haskell ( see http://code.google.com/p/fsharp-typeclasses/ for a real attempt to do that ).
Functions in the prelude are in the FSharpx namespace, so you still have to opt-in with open FSharpx.

flip, curry, uncurry may have originated in Haskell (have they? no idea) but by now I think they're pretty standard especially among ML-like languages, e.g. SML: https://bitbucket.org/seanmcl/imogen/src/3d56401cf034/src/std/fun.sml and the OCaml Batteries project also makes them part of the pervasives module: http://ocaml-batteries-team.github.com/batteries-included/hdoc/BatPervasives.html . Even in F# people wonder sometimes why things like flip aren't standard: http://techneilogy.blogspot.com.ar/2010/06/flip-operator-for-f.html .

As you know they're quite useful for point-free style, and I agree that overusing them can lead to readability issues. But even without them it's possible to abuse point-free just by overusing the standard >> (been there, done that!). And still sometimes point-free is appropriate, and combinators like flip, curry, etc, IMHO can make things more readable, not less.

As usual, it's a matter of balance. People can also use Option.get to abuse options, or abuse refs/mutables and lose the functional benefits of F#, or abuse reflection and lose the benefits of static typing.

Cheers,
Mauricio

[jackfoxy]

I've opened DataStructure organization on its own issue thread #169 to pull together several threads on this topic.

[7sharp9]

@dsyme I agree on the flip and curry functions and I also feel like the monadic operators along with the symbolics add a lot of confusion too. When I look in there I don't want to have to dig out my Haskell reference book to figure things out.

[mausch]

@7sharp9 nobody's forcing you to use flip/curry/etc... you can safely ignore them if they're not your cup of tea.

About monadic operators, the ones I use the most already have named aliases, and sometimes I use them by name and sometimes by symbol. There's an open issue about giving named aliases to the other operators #52 , you're welcome to join the discussion there. Again, if you're not comfortable using monadic operators, just don't use them. All libraries have parts or features that many people don't use often, I don't see why that's an issue particularly with FSharpx.

[ghost]

Hi Maurcio,

Leaving these operators out was a deliberate modification of the 1980s Edinburgh ML programming model made by OCaml in the 1990s and followed by F# (they are not present in the OCaml prelude either). So adding them back into a prelude is definitely altering a deliberate design decision of the language/library designers to avoid the code obfuscation that inevitably arises from having them in a core prelude. Likewise monadic and symbolic operators.

For this reason I'd say they should be removed (or at least put elsewhere than "open FSharpx"). That will help FSharpx be acceptable as a standard F# programming library for a broad range of F# programmers.

It doesn't matter that much as I thnk everyone knows that having these operators globally available across a team of programmers does not make for good F# coding. This means people won't use them much. But the point is that I don't think you want FSharpx to have a reputation for promoting "bad" or "non-standard" F# coding, or you at least want those operators to be more opt-in than "open FSharpx" implies.

Totally point-free programming is relatively rare in F# and it is simple to define operators like these when they are needed within an assembly.

Kind regards
don

[ovatsus]

Maybe we could move the prelude to FSharpx.Compatibility.ML? While I do find a couple of them handy, I remember that the first time I looked at FSharpx, all of that stuff looked scary

[ovatsus]

Sent pull request 172

[mausch]

@dsyme That's a very interesting reference, but I think it only applies to the design of standard libraries. I wouldn't include these operators in a standard library either, it's probably too much of a commitment, it defines the language. But that's not the case with FSharpx, and so the goal is to create a more "batteries-included" library, not to define the core language and primitive functions.
Evidence of this is the Batteries library I mentioned before. Also ocaml-core ( https://bitbucket.org/yminsky/ocaml-core/wiki/Home ) defines flip, curry, uncurry, etc.
Haskell of course goes further, including flip as early as Haskell 1.1 (~1991), const in Haskell 1.2 (1992), curry/uncurry in Haskell 1.3 (1996).

Whenever I read something like curry f, my mind mostly ignores the curry. If I write it explicitly: fun a b -> f(a,b), I have to keep track of the two parameters and check that they're applied in the same order and not flipped. This is a concrete improvement in readability.

Monadic operators in FSharpx require a stronger opt-in, e.g. you need to open FSharpx.Option to get >>=. I think that's pretty safe. Again, for comparison, both Batteries and ocaml-core define >>=. Also Scalaz. It's pretty clear to me that by now monads and its operators have far transcended Haskell and are becoming commonplace.

Finally, it gets really tedious having to type in the definition of basic operators like flip too often, so people will put that in a library. If you look on the net you can see many people defining these operators in F#. I really think it's just best if it's in a hopefully widely used library like FSharpx.

@ovatsus When we all started learning F# it all looked scary and incomprehensible. I then started learning Haskell and it was the same thing all over again. Most of the time, "looking scary" just means that something is unfamiliar, but that doesn't imply it's bad.

Perhaps we could just put a warning in the xml docs of some of the functions in the prelude. For example, Batteries says about flip: "Don't abuse this function, it may shorten considerably your code but it also has the nasty habit of making it harder to read" https://github.com/ocaml-batteries-team/batteries-included/blob/master/src/batPervasives.mli#L463-L467

[ovatsus]

What I was trying to say is that simply by opening FSharpx there's a bunch of scary looking stuff that becomes available that might potentially scare new users. I'm a big fan of some of the monads, but F# really needs more adoption, and one of the best magic tricks @dsyme has done with F# was to call them computation expressions instead of using the "M word" :)

[mausch]

@ovatsus Actually, computation expression aren't strictly related to just monads: you can encode monoids, monadplus, or entirely unrelated things (I saw a computation expression today wrapping COM that didn't look monadic). But yeah, there seems to be a trend among PL designers to avoid naming monads ("warm, fuzzy things!").

I don't agree with the "big, scary library" argument. I wrote a short post about this a few days ago: https://plus.google.com/106809220438980646520/posts/CzKA2AN7ktR
Ultimately, you can always choose whether to use some feature in FSharpx to solve your problem or not. The great thing about F# is that it doesn't force you to use monads or being purely functional, so you can always achieve a solution by other means. But if you do choose to use a monad, then FSharpx saves you from having to reinvent it.

[DanielFabian]

@mausch I think the argument towards trying to avoid a "big, scary library" is more towards the "scary" than the "big". If you build a library that implements all kinds of monads and uses the operators in the same names and contexts as they originated in research, you might make it look like you want to focus on academic issues which would be counterproductive for a language and / or library that is supposed to encourage wide-spread use.

Therefore for instance I very much agree with @ovatsus that the computation expressions (or workflows) being called thus, is a good thing, because people have an intuitive understanding of a "computation" and even more so of a "workflow".

Basically, in my opinion, naming of components, functions, operators, etc. should as closely as possible reflect your intuition. Of course, you can't always succeed, but trying to follow the principle of least astonishment is very important if you want to have maintainable and correct code (or libraries, that are easy-ish to use).

[ghost]

@mausch Unfortunately, in the current design of FSharpx, "Monoid" and "ISemigroup" are not easy to ignore because they are immediately under FSharpx. This means they quickly appear in auto-complete lists.

The usability of auto-complete lists is perhaps the mot important measure of an F# library design. Most people use auto-complete to learn libraries.

For example, when you type "FSharpx" then "DOT" you see concepts like "ISemigroup<T>", "Monoid<T>", "Semigroup" and "Monoid" along with many other types. What's in this auto-complete list is the minimum terminology that people need to know when working with the library (if only to understand what they can safely ignore).

I believe the best solution is to move these operators and types to some other namespace like FSharpx.Functional.

To be honest, the namespacing used in FSharpx is just too shallow, a common problem for libraries at this stage in their evolution (FSharp.Core suffered exactly this problem in the early days - once upon a time everything was in the namespace "FS" :-) ). It looks like it needs to be something like this:

FSharpx.Functional.Abstractions (Monoid, ISemigroup, Lenses)
FSharpx.Functional.Collections (functional data structures)
FSharpx.Interop (C# interop)
FSharpx.Control (StateMachine, STM, TimeMeasurement, Lazy, AsyncSeq)
FSharpx.Data (JSON)
FSharpx.Text (Regex, Strings)

A simple namespace partiitioning like this would make the library much less scary, and may eventually help you to split FSharpx.Core into multiple orthogonal packages.

Kind regards
Don

p.s. I think the arguments used above do apply to the current prelude in FSharpx as much in FSharp.Core. "Batteries included" should not mean "obfuscated code now made easier to author by default" :)

[jackfoxy]

@dsyme Your argument from "FSharpx" then "DOT" is compelling. It is time for a better namespacing and IMO your latest proposed list of namespaces should be the starting point. To move forward the community should hammer out the next iteration in the namespace evolution. Should we consider, for example, FSharpx.Mutable.Collections?

[DanielFabian]

@jackfoxy Namespaces should go from general to specific, e.g. System.Collections.Generic or System.Collections.Specialized. So I'd suggest to have Mutable after Collections.

Could we export the code comments somehow into the wiki as @dsyme suggested? That would also make it easier to cluster the functionality, i.e. classes, interfaces, modules, etc. By having a list or tree of the functionality, we could judge better, what namespaces each piece of functionality belongs to.

[mausch]

@DanielFabian By now monads are far from being a purely academic thing, I've cited three libraries (batteries, ocaml-core, scalaz) that implement monads in languages used in industry (namely OCaml and Scala). Those three libraries implement >>=. In FSharpx there are already named aliases for many of these operators, and the rest of them are being discussed in #52 .

Intuition is personal and not static, it's something you develop over time, with practice and experience. Saying that something "reflects your intuition" actually means that it's intuitive for a particular person in a particular point in time. So I find it hard to use as an argument to generalize things. For example, just three years ago I had absolutely no intuition for monads. I then started learning and gained some understanding, but not quite an intuition. I now have some intuition about monads, i.e. I can more quickly identify monad uses and can read and reason about monadic code faster. But I know I have still much to learn.

Likewise, physicists may develop an intuition for Fourier transforms, and electrical engineers for circuit analysis (see https://plus.google.com/107913314994758123748/posts/AAcy4uwVEz5 ).

The principle of least astonishment IMHO is more about keeping things coherent, uniform. Not about picking any particular names that match anyone's personal intuition, as long as those names are uniform, systematic.

@dsyme That doesn't quite match how I work. In a typical scenario I'm writing some code to get some task done and think "I need a function x -> y / data structure / etc here. This looks generic enough, I bet someone's written this before". Then I look around the web for an implementation. If I find it in an existing library, I download and reference the library. I read some documentation or blog posts, sample code about this library. I open whatever namespaces I need to open to get to the types shown in the blog posts/documentation and write the code to get the task done. The library may have lots of other features, but I don't need them right now so I just ignore them until I need them.

It doesn't match how I learn, either. Say I need to learn about OAuth to use some web API from a .NET client. I find that DotNetOpenAuth implements OAuth in .NET. I wouldn't learn OAuth by downloading DotNetOpenAuth and typing "DotNetOpenAuth." and seeing what's in there. I wouldn't understand anything because I don't understand how OAuth works. Even if I knew OAuth, I don't know a priori how OAuth concepts are mapped in DotNetOpenAuth. Likewise, typing "FSharpx." and seeing a Monoid type won't explain anyone what a monoid is or how to use it. Moving Monoid to FSharpx.AbstractStuff doesn't help learning about monoids either.

The idea behind a single "open FSharpx" (I borrowed this from Scalaz) is that if you know what you need, you can just use it without having to search for the namespace (it also doesn't help that VS + F# lacks autocorrection features). Anything one doesn't understand about FSharpx one can safely ignore, especially since FSharpx is quite wide in features. This is even valid for most libraries I think.

About FSharp.Core, it's interesting that F# implicitly opens many namespaces ( http://msdn.microsoft.com/en-us/library/dd393787.aspx ), essentially creating a flatter default namespace in practice. Having to open those namespaces explicitly would make the language more cumbersome to use overall. I think it's a similar case with FSharpx.

---

Now, it seems pretty clear that many in the community want to split FSharpx in many namespaces or packages (I had a very similar discussion with Tomas Petricek a few months ago), and as a community project there are always compromises and we should reach a consensus even if I don't fully agree. I certainly don't want to come off as stubborn or arrogant (too late? :) ), so if most people think that splitting is for the best, let's do it. Lots of people are getting involved in FSharpx and that's very exciting, I certainly don't want to be the one that stagnates it somehow!

I'd cut the "Functional" from the namespaces since the library is mainly about functional constructs. I also like @DanielFabian's suggestion of adding Mutable to the namespace of mutable collections.

So how about:

• FSharpx.Abstractions
• FSharpx.Collections
• FSharpx.Collections.Mutable (I think we only have one of these now)
• no idea about the rest.

Something I'd really try to avoid is to put flip,curry,uncurry etc in a namespace called 'compatibility', it sounds like legacy. I'd even prefer something like FSharpx.FunctionFunctions.

Also I apologize for hijacking the conversation in a totally unrelated issue which was originally about async in Powerpack, and thanks for reading :)

Cheers,
Mauricio

[7sharp9]

@mausch I think the flip, curry and uncurry have a home here, I have used flip a couple of times recently, admittedly when using with another API.

Fsharpx.FunctionFunctions is a monstrosity please don't :)

Im sure some of the issues people have described would be mitigated with more comments in the API with some more examples etc.

I agree about splitting around the separation of concerns, where that point is I couldn't say without a bit of analysis on the whole API. I wish there were some dependency analysis tools for F#.

Oh and the collection namespaces suggested seem about right too.

[ovatsus]

Another -1 for FSharpx.FunctionFunctions :)
I think rethinking some of the FSharpx namespaces will help, but that doesn't necessarily need to mean that we have to split everything into a lot of namespaces, I think just two or three will be enough. Currently there are some strange things namespace-wide. For example, the built-in Async stuff is in Microsoft.FSharp.Control. We have some extensions in FSharpx and others in FSharp.Control (and the ones from the powerpack I recently added are in the Microsoft.FSharp.Control). I personally think that the extensions to Seq, Async, Map, etc should be in original namespace so that when we reference FSharpx they are immediately available. And most of the stuff could be directly under FSharpx. But would be nice a separate FSharpx.Collections for all new collections, both mutable and immutable, and that the type providers also to be under a different namespace instead of directly under FSharpx.
Probably we should start by enumerating all that's in FSharpx and it's namespace, that will probably also well getting some documentation started

[7sharp9]

@dsyme autogen documentation leaves a lot to the imagination. A few examples of the common usages would be key to a mass adoption. When confronted with a new API the last thing you want to do a hack about trying to figure out how it works.

Maybe new features added to this library should come with good examples use cases?

[ghost]

@mausch I think that's an OK starting point for namespacing.

"Abstractions" in particular would help a lot in addressing the concerns here.

"FSharpx.Collections.Mutable" seems OK too given FSharpx's design goals - at least it seems ot make sense to put any mutable collections into different namespaces (but do you even have any?)

[ghost]

@ovatsus If it helps, the way we would do this at MS is that someone comes up with a strawman proposal (by doing what you say - listing all the pieces and putting them into namespaces). The proposal would be implemented and reviewed by some people not involved in the implementation (hey, I'm here if you need me, ok? :-)).

When reviewing, the library is normally looked at live in Visual Studio and much of the discussion is looking at the intellisense menus, talking about naming/namespaces and asking "is XYZ needed at all?" questions. Feedback is collected from the review process and the process iterates until convergence. It usually takes 2-3 iterations.

This might be backed by qualitative "watch your friend over the shoulder" user studies (e.g. give a friend a script referencing FSharpx and see what they say). These reveal a huge amount and are very cheap to do - it just takes an hour of your time.

We have confidence in the process, i.e. that the end result is more usable for the target audiences, even if the process challenges some of your assumptions and the library may not end up quite how you thought it would. No MS library is perfect, but believe me all of them which have been reviewed this way are a lot more usable than the initial versions proposed by the technology teams :)

[ghost]

@7sharp9 yes, intellisense is more about quickly getting a taste for what's there (or being reminded of it when needed) rather than full docs, examples, tutorials etc.

[jackfoxy]

@ovatsus Separation of mutable and persistent collections is a useful distinction in a library for a functional language.

@dsyme By inspection we appear to have at least 1 mutable in the Collections folder (current FSharpx namespace) and 1 mutable in the DataStructures folder (current FSharpx.DataStructures namespace).

However we name the mutable and persistent namespaces, I suggest a different policy for additions to the collections:

We should spin off (whether through deprecation or a breaking change, TBD) the current DataStructures folder (and possibly add a new folder for mutable) to a different project / namespace, like HTTP, Observable, TypeProviders.

This new project would serve as an incubator and an R&D library of interest to researchers and hackers of DataStructures. New additions should "incubate" until at least 2 or 3 regular contributors / community members are aware of the "newcomer" and approve the quality of the structure and its documentation (can we start requiring some minimal intellisense standards?) We should encourage additions of every variety of every data structure under the sun to this / these namespace(s).

Unique structures like BKTree, IntMap, RoseTree, etc. all belong in the "Core" collections namespaces, but where there are multiple candidates fulfilling a similar role in the incubator, only the "best of breed" should graduate to "Core": 1 Heap, 1 Queue, 1 Deque, 1 CircularBuffer, 1 RandomAccessList, etc.

[7sharp9]

@dsyme RE:

"the CodeDom generator (this is not in good enough shape to integrate - a modified copy now seems to be in use in the MonoDevelop F# bindings anyway)

The edits to the codedom in MonoDevelop look to prettify the output, and also remove the compiler completely. It would be good to get the powerpack and MonoDevelop versions combined although Im not quite sure who would be using it apart from tools that already have codedom output. Im only actually using the compilation part which may be better suited to simply calling FSharp.Build.dll anyway.

[ghost]

Closing this since the main issue referenced (async) is done