Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 161 lines (103 sloc) 9.091 kB
a75659f @chrisdew slow progress
authored
1 Notice
2 ------
3
e0a08ae added docs
chrisdew authored
4 This project is under heavy development and does not yet do anything other than pass some tests, and make the DeltaQL Bootstrap project work.
a75659f @chrisdew slow progress
authored
5
0606aa9 oops
chrisdew authored
6 I'm currently working on the DeltaQL site web application, which will showcase many different queries across a half-dozen pages.
7
a75659f @chrisdew slow progress
authored
8
f2e2bfe first commit
chrisdew authored
9 DeltaQL
10 -------
11
12 Imagine a database (like MySQL or SQLServer) whose 'result sets' automatically update as the underlying data changes. Add to this a facility to extend these 'deltas' all the way to the browser.
13
c2485b4 @wedgeski Fixed typo
wedgeski authored
14 The results of a query are displayed in a web page. They change when (and only when) a change the underlying data alters the result of a particular query. All done *without* 're-executing' the query.
a75659f @chrisdew slow progress
authored
15
f2e2bfe first commit
chrisdew authored
16 Welcome to DeltaQL - no more F5, no more polling the DB.
17
18
0606aa9 oops
chrisdew authored
19 Using DeltaQL
20 -------------
1b08ffe added docs
chrisdew authored
21
46746e2 added docs
chrisdew authored
22 DeltaQL systems begin with Silos. These are simply unordered sets of rows, similar to a database.
1b08ffe added docs
chrisdew authored
23
24 Components, such as Filter, Join, Sort, Union and Head are used to build a query whose results can be displayed in a browser.
25
26 For example:
27
28 var silo = new Silo();
29 var users = silo.filter(function(row) { return row.table === 'users'; });
92d7b36 @chrisdew Update README.md
authored
30 var loggedIn = users.filter(function(row) { return row.loggedIn; });
1b08ffe added docs
chrisdew authored
31 var numLoggedIn = loggedIn.count();
32
a8446a0 added docs
chrisdew authored
33 to send it to a browser, along with all later changes, it's as simple as:
1b08ffe added docs
chrisdew authored
34
35 // render the page
36 app.get('/', function(req, res) {
37 var dqlSess = dql.register(req);
38 dqlSess.add('users', users);
39 dqlSess.add('numLoggedIn', numLoggedIn);
40 res.render('index', { layout: 'layouts/base',
41 title: 'a DeltaQL page'
42 } );
43 });
44
9a9b046 typo
chrisdew authored
45 These components can be joined either in-process or via TCP connections. As soon as an application is suspected of getting too busy for one NodeJS process, the Silo(s) can be moved out to their own process(es).
1b08ffe added docs
chrisdew authored
46
47 A TCP link is a simple as:
48
49 // silo process
50 var silo = new Silo();
51 var users = silo.filter(function(row) { return row.table === 'users'; });
0606aa9 oops
chrisdew authored
52 silo.listen(1234, '127.0.0.1');
53 users.listen(1235, '127.0.0.1');
1b08ffe added docs
chrisdew authored
54
55 // web server process
56 var loggedIn = remoteRSet(1235, '127.0.0.1').filter(function(row) { return row.loggedIn; });
57 var numLoggedIn = loggedIn.count();
58
0606aa9 oops
chrisdew authored
59 As demostrated above, frequently used filters can be placed in the silo process, to help limit unnecessary
60
1b08ffe added docs
chrisdew authored
61 All updates are done to Silos, never to intermediate results, otherwise observering components further 'up' the tree (more accurately a DAG) would not see the change.
62
a75659f @chrisdew slow progress
authored
63
f2e2bfe first commit
chrisdew authored
64 Deliverables
65 ------------
66
67 DeltaQL has five deliverables:
68
a75659f @chrisdew slow progress
authored
69 1. The Bootstrap App. https://github.com/chrisdew/deltaql-bootstrap#readme A minimal NodeJS web app with just one live page - ideal to build upon.
f2e2bfe first commit
chrisdew authored
70
71 2. The Server Library. This is written in Javascript for NodeJS and is used by both your NodeJS web applications and any (optional) separate databases. DeltaQL can be run both in-process and as separate server process(es).
72
73 3. The Browser Library. This runs on the browser and builds on top of KnockoutJS.
74
1b08ffe added docs
chrisdew authored
75 4. The Site App. A working example of a web app built with DeltaQL, possibly demonstrating best practises.
f2e2bfe first commit
chrisdew authored
76
2785bcf removed ebook mention
chrisdew authored
77 All will be available on GitHub under the MIT license.
f2e2bfe first commit
chrisdew authored
78
79
80 Persistance
81 -----------
82
83 DeltaQL operates in memory. It has hooks to add persistence to MySQL, Postgres or any other NodeJS supported database. Use DeltaQL for all your 'live' clients, yet persist your data in a database which you trust.
84
a75659f @chrisdew slow progress
authored
85 The two hooks are a 'save state' (i.e. on controlled shutdown) and a 'delta log' to recover data after a power outage.
86
0606aa9 oops
chrisdew authored
87 Currently the only Persistor is the simple, but inefficient, JsonFilePersistor.
f2e2bfe first commit
chrisdew authored
88
89
90 Origins
91 -------
92
a75659f @chrisdew slow progress
authored
93 When I found myself solving the same 'data freshness' and pub/sub issues in NodeJS for the third time, I recognised the need for a solution to the general case and so created the DeltaQL project.
f2e2bfe first commit
chrisdew authored
94
95
96 Built on the Shoulders of Giants
97 --------------------------------
98
99 DeltaQL uses NodeJS, Express, Socket.IO, Mocha, KnockoutJS and much other uncredited work from the open source community. Thanks to all of those who've made DeltaQL achievable in only tens-of-hours of design and coding.
100
101
102 Jargon
103 ------
104
105 This may be useful if you start diving into the code.
106
107 * Lop - a List OPeration. This is a delta which keeps a two ResultLists in sync.
108 * ResultList - an *ordered* list of rows.
109 * ResultSet - a set of rows.
110 * Row - a JSON object with an 'id' field containing an unique string (often a UUID) and zero or more further fields.
111 * Sop - a Set OPeration. This is a delta which keeps a two ResultSets in sync.
112 * LivePage - any web page containing one or more ResultLists.
a75659f @chrisdew slow progress
authored
113 * Filter - a transform from a ResultSet to a possibly smaller ResultSet
114 * Sort - a transform from a ResultSet to a ResultList
0606aa9 oops
chrisdew authored
115 * Head - a transform from a ResultList to a possibly smaller ResultList
5fa3368 testing bulleted lists
chrisdew authored
116 * Tail - a transform from a ResultList to a possibly smaller ResultList
117
118
119 Shortcommings
120 -------------
121
d904672 testing bulleted lists
chrisdew authored
122 This project is about building something for people to play with and critique in a few tens of hours.
5fa3368 testing bulleted lists
chrisdew authored
123
124 Some of the aspects of the project which could do with major improvement include:
4885741 testing bulleted lists
chrisdew authored
125
5fa3368 testing bulleted lists
chrisdew authored
126 * Testing - there is a test suite, but coverage is very far from 100%
127 * Examples - is would be good to have a whole directory of exmaples, rather than just a couple of example projects.
e0a08ae added docs
chrisdew authored
128 * IE - I'm not putting any effort into making anything work on IE, yet. I expect to support IE8+ later.
0606aa9 oops
chrisdew authored
129 * Persistors - Add code to save state in MySQL, Postgres.
130 * Protocol adapters, E.g.
131 * Adapt an in-process SMTP server makes email available as a Silo.
132 * How about IRC channels as Silos?
133 * Perhaps a DNS server, serving data from a Silo, for locating RemoteRSet's ports and addresses via srv records?
134 * Twitter/RSS feeds as Silos?
135 * Failover remote Silos - at some point there will be need to be master/slave replication for when a remote Silo's hardware breaks. Could instead go with a more fine-grained event-oriented multi-master system.
136 * Webserver Affinity - At the moment the system relies on clients connecting back to the same server when using SocketIO. At the moment I have disabled all awkward transports, such as websockets, as they break this design when used with a load balancer. This needs to be solved, wither by better session affinity or storing dqlSessions in a remote Silo (rather than them being in-process).
5fa3368 testing bulleted lists
chrisdew authored
137 * API documents - that would be nice
0606aa9 oops
chrisdew authored
138 * Use https://github.com/chrisdew/multicast-eventemitter for RemoteRSets after the initial state has been gained via TCP - this would be hugely more efficient on local clusters. (We could add a sequence number to the multicast packets, and trigger a full reload (after a random interval, to stop thundering herds) via TCP if a sequence number was missed.)
139 * Add some geospatial filters, i.e. isWithinPolygon(), or isWithinXMetres().
140 * Add some chronologically aware filters, i.e. arrivedWithinX() and rowFieldNotOlderThanX() - these require setTimeouts to be set to remove rows X after they arrive (or similar condition) - rather than polling. I've done this before, so I know it's practical. I've (due to a bug) once had a result set which introduced a 5 hour delay in sending data to browser, due to a MySQL backlog. NodeJS is very resiliant with regard to callbacks and timeouts.
141 * Time should be milliseconds since epoch UTC, everywhere - let your browser-side code deal with its display (in localtime).
5fa3368 testing bulleted lists
chrisdew authored
142 * Safety - at the moment most updates are done through simple event emitters. These provide no feedback on (for example) failure to write to disk.
143 * Efficiency - E.g.
144 * Pushing Filters Upstream - if a filter function captures no variables, and it has a remote parent, then it can be pushed across a network connection to reduce the number of number of ops sent over the wire.
145
146 These issues are all fixable, given a few weeks of work. I'm not even going to think about that until I know that I'm solving a problem that exists for more than just me.
a75659f @chrisdew slow progress
authored
147
148
149 Licence
150 -------
151
152 DeltaQL is available under the MIT licence:
153
154 Copyright (C) 2012 Chris Dew.
155
156 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
157
158 The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
f2e2bfe first commit
chrisdew authored
159
a75659f @chrisdew slow progress
authored
160 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Something went wrong with that request. Please try again.