StreamSheets - streams as spreadsheets

[stevekrouse]

StreamSheets

In many scattered places in my /journal, I mention the StreamSheets project. I also mention it in /ideas. However until I reorganize this website around ideas (as opposed to time), this will remain a problem. In the meanwhile, I will use this github issue to track my thinking around this project.

My goal is to have a goal

This project has very clear goals, because I created it after listening to a talk by Alan Kay in which he implores us to making technology that's neatsolve problems that exist, regardless of the technology that exists today. This means that if you have to create a whole host of more fundamental technology to accomplish your main goal, do it. If you merely look at the technologies close at hand, you will only come up with solutions close at hand. And thus you will only arrive at incremental solutions.

And, in today's day given all the programmers out there, chances are if you are making an incremental improvement, someone's already done it. The only way to produce novel work it so solve fundamental problems with a novel solution where many of its sub-components do not yet exist.

Goal

There are few ways to state the goal of this project but they all amount to approximately the same thing.

1. Create a programming environment such that the things built in it are editable by their end users.

Imagine clicking an "edit" button on an app like facebook, changing the color of a button (or something more complex), and publishing it to a place where your friends (or the developers of facebook) could accept this change, all without leaving facebook.

This is impossible for two reasons:

1. Facebook is closed source. This means that a) anyone with a good idea of how to improve it can't besides minor changes through hacks like chrome extensions, and b) anyone that wants to customize facebook for their particular use case cannot -- we all get the same product. More on solving the open source problem below.

2. Even if facebook was open source, it would be incredibly hard for 99% of its users to navigate through the files and folders that make up its code, find the relevant places in the codebase, figure out how to properly change it, install the proper software and configure it to be able to compile the code with their changes, and then deploy these changes to a place where they can use them, not to mention to a place where their friends or the developers of facebook could adopt them if they wanted to.

2. Allow users to "peel back layers" (or "pop the hood")

The way I originally visualized allowing users to do this is by letting them "peel back" various layers of their application to be able to see what's underneath. Imagine that you could click and drag the top-right-corner of any application, peeling back the topmost layer of the interface, exposing the internals of the application one layer of abstraction below. Or if you're just curious about one component of the page, you can just peel back the layer for that component.

The thing is that this lower level that you're now looking at is an interface in its own right. You can change various pieces of data, or the way the data is hooked up. Then you can relayer the layer that you peeled off back on again to see what your changes look like.

In theory, you could keep peeling back layer after layer, drilling deeper and deeper into how the application works, until you get to the core data structures.

Another way to think about this is that it's similar to how you can pop the hood of a car, tweak things, and then pop it back down to see the effect of your changes. However I don't love this metaphor because only mechanics really know how to tweak things, and I'd like to make it so that anyone should be able to tweak things.

This metaphor directly solves the problem of not even knowing where in an open source project the relevant code you need to edit is to cause the change you desire to make. Instead of blinding hunting around in files or folders (or reading outdated READEME.md files), you can simply click on the widget you wish to edit and begin editing it.

3. Make all software open source.

Some look at the price for developer tools (often $0) and say that "nobody values developer tools." However, that is not the case. The price for anything is influenced not only by demand but also by supply. I would argue that developers care deeply about their tools (have you seen them talk about their text editors?) so the demand is high. The reason the price is almost nothing is that supply is so high.

Why is the supply high? Because developers can make their own tools. And when people can scratch their own itch, they tend to scratch it. In other words, anyone who creates developer tools is competing with all developers, so over time developers will keep undercutting each other on price until that price drops to $0. (Of course, there's still space in the market to have a large team which produces higher quality software than lone developers can produce (like Jetbrains) and charge for it, because the only people who'd be able to undercut you are other large teams which also need to pay expenses and thus can't give away software for free.)

However the software in other industries is often very expensive, partially because the people who would use such software couldn't make it themselves. But what if they could?

An open source project will only be successful in the long run (without outside financing) if the community that uses the tool can also contribute to its development, such as the Linux community. I think about this as the "open source intersection" of a project, the amount of people who use a project that can also contribute to its development. If you can get this number (and I think it's an absolute number, not a percentage but I need to think on this more) high enough, a project can continuously improve indefinitely by its users for free (although I have nothing against paying people if the community is able raise funds like CycleJS does through OpenCollective.)

However, as long as traditional coding languages and paradigms are used, where it takes professional programmers like myself hours, potentially days to get acquainted with even the best maintained codebases, this number will remain quite small for all but developer-facing projects -- and even these projects won't be maintained well enough because it takes hours and hours and hours for professional developers to get acquainted with a new codebase. I know because I've spent dozens of hours trying to submit patches to open source projects, only to overcome with frustrating after trying to figure out where in the codebase I need to make my change or how to even get the code to compile. Making things better is fun. Making things better that you and your friend use is sublime. People would love to do it if they could. And then software would be better and cheaper for everyone.

In conjunction with IPFS and other peer to peer blockchain type foundations, this could allow the world to create software collaboratively for free, for everyone, forever. This would drastically increase the amount and quality of software at the same time as the price of the software approaches zero. Yes it would be sad to see all those developers put out of a job, but I guess that's why developers are so crazy about getting universal basic income passed 😝

Similar efforts

These goals aren't original. The theme behind them has been inspired largely by Alan Kay et al.

Hypercard

People often talk about how Hypercard accomplished similar ones but I don't know much about it.

Alan Kay's STEPS Project

Alan Kay's STEPS project had the goal of creating better foundations for software by creating an entire operating system and GUI engine in 20,000 lines of readable and editable code. Ironically enough they go about it using FRP and streams, which is currently how I am thinking about solving this problem.

The View Update Loop Problem

Ok, so now that we're clear on the goal, let's talk about the problem. We as a society have largely solved the problem of static HTML and CSS website creation though WYSIWYG editors like Weebly or Wix. However, we are still trying to solve this problem for designing dynamic interfaces, although there's been a lot of amazing development on this front in recent years, from JQuery to AngularJS to ReactJS to CycleJS and Elm.

Here's the problem with dynamic interfaces: "the view update loop problem," the fact that interacting with the interface will affect that very interface. If you were to draw this, it would look something like: data -> page -> events -> where this last arrow wraps back around to modify the data, which modifies the page, which produces events, which goes round and round.

Things are much simpler in a tool that produces a static view, such as static website creator like Weebly or Wix mentioned above, or even in a spreadsheet like Excel. In Excel there are no events that modify data. The only way to modify data is to directly modify the data in a cell.

There are lots of ways to solve this problem, as many as there are MV* frameworks, of which there seemingly as many as there are developers. However, starting with ReactJS, the community has moving more and more towards FRP.

FRP (functional reactive programming)

Firstly, I should caveat this section with the qualification that I am no reactive programming expert (yet!) so I'd appreciate a comment if I get something wrong.

Secondly, I should explain that FRP takes a while to wrap one's head around. It took me days of effort to grok ReactJS, reading every page of its documentation multiple times -- and that's coming from both a web programming and functional programming background. Please allow extra time if you're new to either web programming or functional programming. It also took me days to get familiar with Elm -- again, that's coming off of years of experience with Haskell, so expect it to take a while if you're new to functional programming. Finally, I also had to spend weeks wrapping my head around CycleJS -- even after learning ReactJS and Elm, partially because explicit stream-based programming (as opposed to implicit stream programming in ReactJS and Elm) is brain-bending.

FRP was originally created by Conal Elliott and Paul Hudak in this paper (which I haven't yet read because I'm the worst but will read soon I promise!) in 1997. The main idea behind FRP is that it's declarative (as opposed to imperative).

Declarative

I've always had trouble with this word. It's definitely an important word, but it's one that I think is hard to describe. [Here's how Evan Czaplicki describes it in his thesis about Elm](Elm: Concurrent FRP for Functional GUIs):

The term declarative makes a distinction between the what and the how of programming. A declarative language allows you to say what is displayed, without having to specify exactly how the computer should do it. With functional reactive programming, many of the irrelevant details are left to
the compiler, freeing the programmer to think on a much higher level than with most existing GUI frameworks.

The term declarative is important because most current frameworks for graphical user interfaces are not declarative. They mire programmers in the many small, nonessential details of handling user input and modifying the display. The declarative approach of functional reactive programming makes it quick and easy to specify complex user interactions. FRP also encourages a greatly simplified approach to graphical layout. As a result, FRP makes GUI programming much more manageable than with traditional approaches.

However, I think this article by Haoyi best explains the distinction between declarative and imperative. It uses the traditional cooking recipe metaphor to illustrate the point. The regular cooking recipie that we're all used to is the classic imperative program:

Before I read this article, I though that a declarative recipe for a chocolate cake was "make me a chocolate cake" and the chief figured out all the lower level details for you. This is the core problem about defining a word like "declarative": who gets to decide which details are "lower level"? The classic example of a "declarative" language is SQL, because it describes what data is desired, not the actual searching, sorting and filtering steps needed to be taken out to get that data. However that's just declarative relative to a programmer who knows about searching, sorting and filtering algorithms because she learned them in class. To a business person, the declarative query is "give me a barchart of widget sales over time." The SQL query and chart graphing code are mere details that an engineer needs to figure out and are too low level for a business person. In fact any non-programmer who specifies what an end-product needs to look like is using spoken-languages and mockups as their declarative language to describe what they want -- programmers are then the compilers that convert this into the lower level code.

However, in the article Haoyi, who's really explaining functional programming specifically, not declarative programming, makes the distinction that "the core of Functional Programming is thinking about data-flow rather than control-flow", which allows you to more clearly understand your program and be able to answer questions such as "If I have two people to make this tiramisu, which parts can be done in parallel?" and "What if my eggs haven't arrived? Which steps can I do first before the eggs turn up?"

In summary, declarative is a strange word, partially because it's a relative word, but in the context of programming, declarative means not focusing on control flow and yes focusing on data flow.

Another way that I like to think about declarative is that things are declared once and only once and nothing anywhere else can change them ever. TODO continue here

FRP goes mainstream

However it wasn't until ReactJS that FRP was adopted as a major development paradigm. (This could be blatantly false. I was just a wee child during this time so wasn't around to see what was going on. Please correct me if FRP was a mainstream paradigm before React.)

TODO continue here

CycleJS takes FRP extreme

TODO explain how this isn't the case in STEPS and how it's still not totally pure yet (random and screen size)

Elm is also relevant

TODO

Streams as Spreadsheet

TODO