-
-
Notifications
You must be signed in to change notification settings - Fork 3
/
type-the-world.scroll
182 lines (127 loc) · 14.4 KB
/
type-the-world.scroll
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
date 2020-01-20
tags All Programming Thinking Scroll
title Type the World
header.scroll
printTitle
thinColumns 2
In this post I briefly describe eleven threads in languages and programming. Then I try to connect them together to make some predictions about the future of knowledge encoding.
dateline
This might be hard to follow unless you have experience working with types, whether that be types in programming languages, or types in databases, or types in Excel. Actually, this may be hard to follow regardless of your experience. I'm not sure I follow it. Maybe just stay for the links. Skimming is encouraged.
endSnippet
# First, from the Land of Character Sets
Humans invented characters roughly 5,000 years ago.
https://en.wikipedia.org/wiki/List_of_languages_by_first_written_accounts 5,000 years ago
Binary notation was invented roughly 350 years ago.
https://en.wikipedia.org/wiki/Binary_code Binary notation
The first widely adopted system for using binary notation to represent characters was ASCII, which was created only 60 years ago. ASCII encodes little more than the characters used by English.
https://en.wikipedia.org/wiki/ASCII 60 years ago
In 1992 UTF-8 was designed which went on to become the first widespread system that encodes all the characters for all the world's languages.
https://en.wikipedia.org/wiki/UTF-8 UTF-8 was designed
*For about 99.6% of recorded history we did not have a globally used system to encode all human characters into a single system. Now we do.*
# Meanwhile, in the Land of Standards Organization
Scientific standards are the original type schemas. Until recently, Standards Organizations dominated the creation of standards.
https://en.wikipedia.org/wiki/Standards_organization Standards Organizations
You might be familiar with terms like meter, gram, amp, and so forth. These are well defined units of measure that were pinned down in the International System of Units, which was first published in 1960.
https://en.wikipedia.org/wiki/International_System_of_Units International System of Units
The International Organization for Standardization (ISO) began around 100 years ago and is the organization behind a number of popular standards from currency codes to date and time formats.
https://www.iso.org/popular-standards.html popular standards
https://www.iso.org/iso-4217-currency-codes.html currency codes
https://www.iso.org/iso-8601-date-and-time-format.html date and time
*For 98% of recorded history we did not have global standards. Now we do.*
# Meanwhile, in Math Land
My grasp of the history of mathematics isn't strong enough to speak confidently to trends in the field, but I do want to mention that in the past century there has been a lot of important research into type theories.
https://en.wikipedia.org/wiki/Type_theory type theories
In the past 100 years type theories have taken their place as part of the foundation of mathematics.
https://en.wikipedia.org/wiki/Foundations_of_mathematics foundation of mathematics
*For 98% of recorded history we did not have strong theories of type systems. Now we do.*
# Meanwhile, in Programming Language Land
The research into mathematical type and set theories in the 1900's led directly into the creation of useful new programming languages and programming language features.
From the typed lambda calculus in the 1940's to the static type system in languages like C to the ongoing experiments of Haskell or the rapid growth of the TypeScript ecosystem, the research into types has led to hundreds of software inventions.
https://en.wikipedia.org/wiki/Simply_typed_lambda_calculus typed lambda calculus in the 1940's
https://en.wikipedia.org/wiki/C_(programming_language C
https://www.haskell.org/ Haskell
https://www.typescriptlang.org/ TypeScript ecosystem
In the late 1990's and 2000's, a slew of programming languages that underutilized innovations from type theory in the name of easier prototyping, like Python and Ruby and Javascript, became very popular. For a while this annoyed programmers who understood the benefits of type systems. But now they too are benefiting, as there is a bigger demand for richer type systems now due to the increase in the number of programmers.
*95%+ of the most popular programming languages use increasingly smarter type systems.*
# Meanwhile, in API Land
Before the Internet became widespread, the job of most programmers was to write software that interacted only with other software on the local machine. That other software was generally under their control or well documented.
In the late 1990's and 2000's, a big new market arose for programmers to write software that could interact over the Internet with software on other machines that they had no control of or knowledge about.
At first there was not a good standard language to use that was agreed upon by many people. 1996's XML a variant of SGML from 1986, was the first attempt to get some traction for this job. But XML and the dialects of XML for APIs like SOAP (1998) and WSDL (2000) were not easy to use. Then Douglas Crockford created a new language called JSON in 2001. JSON made web API programming easier and helped create a huge wave of web API businesses. For me this was great. In the beginning of my programming career I got jobs working on these new JSON APIs.
https://en.wikipedia.org/wiki/XML XML
https://en.wikipedia.org/wiki/Standard_Generalized_Markup_Language SGML
https://en.wikipedia.org/wiki/SOAP SOAP (1998)
https://en.wikipedia.org/wiki/Web_Services_Description_Language WSDL (2000)
https://en.wikipedia.org/wiki/Douglas_Crockford Douglas Crockford
https://www.json.org/ JSON
The main advantage that JSON had over XML was simple, well defined types. It had just a few primitive types—like numbers, booleans and strings—and a couple of complex types—lists and dicts. It was a very useful collection of structures that were important across all programming languages, put together in a simple and concise way. It took very little time to learn the entire thing. In contrast, XML was "extensible" and defined no types, leading to many massive dialects defined by committee.
*For 99.8% of recorded history we did not have a global network conducting automated business transactions with a typed language. Now we do.*
# Meanwhile, in SQL Land
When talking about types and data one must pay homage to SQL databases, which store most of the world's structured data and perform the transactions that our businesses depend on.
https://en.wikipedia.org/wiki/SQL SQL
SQL programmers spend a lot of time thinking about the structure of their data and defining it well in a SQL data definition language.
https://en.wikipedia.org/wiki/Data_definition_language data definition language
Types play a huge role in SQL. The dominant SQL databases such as MySQL, SQL Server, and Oracle all contain common primitives like ints, floats, and strings. Most of the main SQL databases also have more extensive type systems for things like dates and money and even geometric primitives like circles and polygons in PostgreSQL.
https://www.postgresql.org/docs/9.5/datatype-geometric.html circles and polygons
Critical information is stored in strongly typed SQL databases: Financial information; information about births, health and deaths; information about geography and addresses; information about inventories and purchase histories; information about experiments and chemical compounds.
*98% of the world's most valuable, processed information is now stored in typed databases.*
# Meanwhile, in the Land of Types as Code
The standards we get from the Standards Organizations are vastly better than not having standards, but in the past they've been released as non-computable, weakly typed documents.
There are lots of projects that are now writing schemas in computable languages. The Schema.org project is working to build a common global database of rich type schemas. JSON LD aims to make the types of JSON more extensible. The DefinitelyTyped project has a rich collection of commonly used interfaces. Protocol buffers and similar are another approach at language agnostic schemas. There are attempts at languages just for types. GraphQL has a useful schema language with rich typing.
http://schema.org/ Schema.org project
https://json-ld.org/ JSON LD
https://github.com/DefinitelyTyped/DefinitelyTyped DefinitelyTyped
https://developers.google.com/protocol-buffers Protocol buffers
https://capnproto.org/ similar
https://www.typedefs.com/ attempts
https://graphql.org/ GraphQL
*100% of standards/type schemas can now themselves be written in strongly typed documents.*
# Meanwhile, in Git Land
Git is a distributed version control system created in 2005.
https://en.wikipedia.org/wiki/Git Git
Git can be used to store and track changes to any type of data. You could theoretically put all of the English Wikipedia in Git, then CAPITALIZE all verbs, and save that as a single patch file. Then you could post your patch to the web and say "I propose the new standard is we should CAPITALIZE all verbs. Here's what it would look like." While this is a dumb idea, it demonstrates how Git makes it much cheaper to iterate on standards. Someone can propose both a change to the standard and the global updates all in a single operation. Someone can fork and branch to their heart's content.
*For 99.9% of recorded history, there was not a cheap way to experiment and evolve type schemas nor a safe way to roll them out. Now there is.*
# Meanwhile, in Hub Land
In the past 30 years, central code hubs have emerged. There were early ones like SourceForge but in the past ten years GitHub has become the breakout star. GitHub has around 30 million users, which is also a good estimate of the total number of programmers worldwide, meaning nearly every programmer uses git.
https://sourceforge.net/ SourceForge
http://github.com/ GitHub
In addition to source code hubs, package hubs have become quite large. Some early pioneers are still going strong like 1993's CRAN but the breakout star is 2010's NPM, which has more packages than the package managers of all other languages combined.
https://cran.r-project.org/ CRAN
http://npmjs.com/ NPM
Types are arbitrary. The utility of a type depends not only on its intrinsic utility but also on its popularity. You can create a better type system—maybe a simpler universal day/time schema—but unless it gains popularity it will be of limited value.
Code hubs allow the sharing of code, including type definitions, and can help make type definitions more popular, which also makes them more useful.
*99% of programmers now use code hubs and hubs are a great place to increase adoption of types, making them even more useful.*
# Meanwhile, in Semantic Web Land
The current web is a collection of untyped HTML pages. So if I were to open a web page with lots of information about diseases and had a semantic question requiring some computation, I'd have to read the page myself and use my slow brain to parse the information and then figure out the answer to my semantic question.
The Semantic Web dream is that the elements on web pages would be annotated with type information so the computer could do the parsing for us and compute the answers to our semantic questions.
https://en.wikipedia.org/wiki/Semantic_Web The Semantic Web dream
While the "Semantic Web" did not achieve adoption like the untyped web, that dream remains very relevant and is ceaselessly worked upon. In a sense Wolfram Alpha embodies an early version of the type of UX that was envisioned for the Semantic Web. The typed data in Wolfram Alpha comes from a nicely curated collection.
https://www.wolframalpha.com/ Wolfram Alpha
https://datarepository.wolframcloud.com/ curated collection
While lots of strongly typed proprietary databases exist on the web for various domains from movies to startups and while Wikipedia is arguable undergoing gradual typing, the open web still remains largely untyped and we don't have a universally accessible interface yet to the world's typed information.
http://imdb.com/ movies
https://www.crunchbase.com/ startups
https://en.wikipedia.org/wiki/Gradual_typing gradual typing
*99% of the web is untyped while 99% of the world's typed information is silo-ed and proprietary.*
# Meanwhile, in Deep Learning Land
Deep Learning is creeping in everywhere. In the past decade it has come to be the dominant strategy for NLP. In the past two years, a new general learning strategy has become feasible where models learn some intrinsic structure of language and can use this knowledge to perform many different language tasks.
https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf general
https://github.com/google-research/ALBERT feasible
One of those tasks could be to rewrite untyped data in a typed language.
*AI may soon be able to write a strongly typed semantic web from the weakly typed web.*
# Tying All These Threads Together
I see a global pattern here that I call the "Type the World" trend. Here are some future predictions from these past trends.
- We will always have creative, ambiguous, untyped human languages where new ideas can evolve freely
- In the future great new ideas from the untyped realm will be adopted faster by the typed realm
- Nearly all transactions in business and government will be in typed languages
- Someone will invent a wildly popular new domain specific language(s) for type definitions
- All the popular standards will be ones written in these new and improved TypeDSLs
- Git—or git like systems—will be used to store both the TypeDSLs and the typed data
- TypeHubs will arise hosting these widely used type schemas
- Programmers will get their types from TypeHubs regardless of which programming language they use
- Deep learning agents will be used to rewrite the web's untyped data into typed data
- Deep learning agents will be used to improve type schemas
- Human editors will review and sign off on the typing work of the deep learning agents
- Silo-ed domain specific standards will merge into one or a handful of global monolithic type systems
The result of this will be a future where all business, from finance to shopping to healthcare to law, is conducted in a rich, open type system, and untyped language work is relegated to research, entertainment and leisure.
While I didn't dive into the benefits of what Type the World will bring, and instead merely pointed out some trends that I think indicate it is happening, I do indeed believe it will be a fantastic thing. Maybe I'll give my take on why Type the World is a great thing in a future post.
footer.scroll