Permalink
Switch branches/tags
Find file
Fetching contributors…
Cannot retrieve contributors at this time
182 lines (139 sloc) 8.25 KB
Regression:
Code introduced during FindBugs refactoring makes \f hang the console.
most important new features:
-- range intervals over rows, not just time
-- chunking on the fly (not rolling windows)
-- ability to have these intervals start exactly on the hour/minute/etc
-- persistent flow metastore
-- JSON event input format
-- dependencies: LIST<T>, MAP<KT,VT>
-- multi-threading, distribution, scalability
-- udf api
Types:
-- Add CHARACTER type which is a single char.
-- need Map<T1, T2> types
-- more list functions:
prepend/append(list<t> lst, t val) -> returns lst @ [val] ?
reverse(list<t> lst) -> reverses the order of elements in lst
map_fn(fn<t> f, list<t> lst) -> applies f to each value in lst and returns the result list.
exists(fn<t> f, list<t> lst) -> true if f returns true for some element of lst
foldl(fn<x, t> f, x initial, list<t> lst) -> folds f over elements of lst, using
initial as the first accumulator value.
-- BINARY type should be able to specify encoding when converting to/from STRING.
Aggregation:
-- unit test, unit test, unit test, baby.
... for a bit later:
-- Agg funcs cannot be used in broader expressions (Cannot select 1 + COUNT(x)).
In cases where they are used like this, assign the function's output a new
anonymous name, promote the aggfunc to the aggregate functions list,
and replace the call to the agg func in the expression with an IdentExpr to
look up the value of this function's output.
-- Allow use of WINDOW clauses inside AliasedExprs, rather than having a single
global GROUP BY?
-- Implement SQL:2003 windowing functions like LEAD, LAG, RANK. See slides at
http://wwwlgis.informatik.uni-kl.de/cms/fileadmin/courses/SS2008/NEDM/RDDM.Chapter.06.Windows_and_Query_Functions_in_SQL.pdf
-- Also new scalar functions listed there.
-- Allow additional expr arguments (where isConstant() is true) to parameterize calls
to aggregate functions. e.g., LEAD(x, 2), LAG(y, 3), etc.
-- CHECK: Do we ever need non-ConstExpr arguments? Do we ever need to aggregate over
two columns at once?
Remote environment stuff:
-- Design question: think about whether operations like ExecEnv.setFlowName() should be
synchronous or async. They're currently the latter -- would users expect
the former?
-- Icing: Operations like \w <streamName> should call a method that resolves a name to
a flowId, and then calls watch() on the flowId.
Joins:
I don't know that the various calls to streamSymbol.getName() are actually
the correct name here -- I think we want the user's alias to get its way
into the various source types.
SelectStmt should support WINDOW clauses at the end.
^-- need to eval them to WindowSpecs and store these bindings for use in joinedsrc
evaluation to hashjoinnodes.
^-- actually, I think this already works (it does for GROUP BY). Add a unit test
to ensure that they work with joins, too.
.. what happens if we use the same stream alias in two different subtrees of
a compound select stmt?
SELECT ... FROM
((SELECT x FROM f JOIN g) AS s1)
JOIN
((SELECT x FROM h AS f JOIN g) AS s2)
Specifically w.r.t. concerns over the lists returned by getSourceNames().. does
this stop at the SELECT level, or does it recursively gather?
I think we're ok for SELECT/JOIN... but I don't know how we will do for predicate
push-down later..
Expressions:
- Refine user_sel rule (or TypeChecker) so that it cannot accept identifier names that
end in "_". (We need these for internal field identifiers in avro.)
Technical debt:
- Write more unit tests as outlined at the bottom of TestSelect.
- Need more tests in TestTypeChecker; see TODO in comments.
- LocalEnv / LocalEnvThread is full of SelectableQueue<Object> because we put
both EventWrappers and ControlOps in different threads. We then typecase on
what we get back. Would be nice if we could create some common interface
with a dispatch() / delegate()-and-handle() model to clean up the big
typecase in the run() method...
- Select.join() will consult its underlying Selectables in a stable order,
which means we might starve later Selectables in the list. We need to
perturb this to ensure fairness.
- The various dag operators all type-parameterized by what sort of nodes
they expect to exist in the dag. They're invoked by calling theDag.reverseBfs(operator),
etc. This is backwards. Each operator is intended to be run in a specific fashion
(bfs, dfs, etc). We should have several operator subclasses; the base specifies
an abstract exec(theDag) method; the subclasses BfsOperator, DfsOperator, etc. then
invoke the right dag.reverseBfs(this), dag.dfs(this), etc.
SQL Features (in rough priority order):
- LIMIT (within a window?)
- [windowed] ORDER BY -- how does this work for overlapping windows?
Bug in CREATE STREAM AS SELECT and Flume node management:
- We currently assume the Flume world is fairly static. If we set up a stream to
read from node 'x', we will create a new local logical node named 'x-receiver'.
If 'x' disappears, we do not delete 'x-receiver'. But if 'x' later reappears,
we never re-connect it to 'x-receiver'. The changes needed to handle this would
be in ForeignNodeConn / EmbeddedFlumeConfig.
** Consider a watchdog thread associated with each receiver. If the upstream
disappears, close the receiver and all associated flows.
** Even better: a watchdog that waits for the upstream node to disappear. If it
then reappears, we re-configure it to again connect down to our x-receiver
in addition to whatever new downstream source it has.
- CSAS creates a local output node for the stream. If the stream gets renamed
using \dname, or dropped (via DROP STREAM), any dependent running flows will
terminate.
** We should not allow dropping a flow associated with CSAS as long as any flows
require that flow to exist upstream.
** More broadly: We should not allow DROP STREAM if any running flows depend on
that stream.
Bugs:
- Quitting is very slow due to the Flume shutdown. Can we improve this?
... it also emits a scary looking error message, that we should suppress for
hygeine's sake.
- Windowed JOIN may cause server crash under undefined conditions.
Features:
- Need ability to parse strings into timestamps.
- Windowing should operate on 'previous n rows' too.
Associate a rowid stamp with every event on input.
Start each stream at 0 for each query, join operates like a "zipper"
- Need ability to run remote physical plan on a set of configured nodes.
- EventParser names/implementations should be accessed through the BuiltInSymbolTable.
- Need a MapFunc API to allow 1-to-many transformations, and LATERAL VIEW syntax.
- Should be able to CREATE STREAM AS FILE and then specify the file format. Right now
we can't practically use avro data in files, because we parse the file itself as
if it is \n-terminated text records.
- Replace jline with libjcsi or a Java readline wrapper
Optimizations:
- ProjectionNode instances with identical input and output schemata should be removed.
- ProjectionNode immediately after NamedSource should be fused.
-- longer term: projection (and filtering) should be pushed up into previous
FlowElements, when we use DAG tiling.
- AvroEventWrapper could be improved by deserializing only the fields necessary
for a given combined set of FE's in advance, knowing the input, output, and also
internal-use schemas. (We may take in <int, string> and emit <int, string> but
only query the int component; we could project onto that as we deserialize, and
then save ourselves the trouble on the string.)
- Expr.eval() should go by the wayside; what we really want is each Expr contributing
a set of three-addr codes to a set of basic blocks that corresponds to the set of
expressions the user wants emitted; then we can perform further optimization on
this like common subexpression elimination, etc. We need to define a RTL that can
handle all this. Then if our opcodes are things like AddIntInt, AddFloatFloat, etc.,
we can dispense with the stored Type instances inside Exprs being required at
run time.