RBQL - RainBow Query Language: SQL-like language. Supports "SELECT" and "UPDATE" queries with Python and JavaScript expressions
Python JavaScript Shell
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
test_datasets
unit_tests
.eslintrc.json
LICENSE
README.md
cli_rbql.js
cli_rbql.py
rbql.js fix Aug 13, 2018
rbql.py
rbql_utils.js more robust quoted split Jan 24, 2018
rbql_utils.py
template.js.raw
template.py.raw
test_all.sh
test_rbql.py
unit_tests.js

README.md

RBQL (RainBow Query Language) Description

RBQL is a technology which provides SQL-like language that supports SELECT and UPDATE queries with Python or JavaScript expressions.

Official Site

Main Features

  • Use Python or Java Script expressions inside SELECT, UPDATE, WHERE and ORDER BY statements
  • Result set of any query immediately becomes a first-class table on it's own.
  • Output entries appear in the same order as in input unless ORDER BY is provided.
  • Input csv/tsv spreadsheet may contain varying number of entries (but select query must be written in a way that prevents output of missing values)
  • Works out of the box, no external dependencies.

Supported SQL Keywords (Keywords are case insensitive)

  • SELECT [ TOP N ] [ DISTINCT [ COUNT ] ]
  • UPDATE [ SET ]
  • WHERE
  • ORDER BY ... [ DESC | ASC ]
  • [ [ STRICT ] LEFT | INNER ] JOIN
  • GROUP BY
  • LIMIT N

All keywords have the same meaning as in SQL queries. You can check them online

RBQL-specific keywords, rules and limitations

  • JOIN statements must have the following form: <JOIN_KEYWORD> (/path/to/table.tsv | table_name ) ON ai == bj
  • UPDATE SET is synonym to UPDATE, because in RBQL there is no need to specify the source table.
  • UPDATE has the same meaning as in SQL, but it also can be considered as a special type of SELECT query.
  • TOP and LIMIT have identical meaning. Use whichever you like more.
  • DISTINCT COUNT is like DISTINCT, but adds a new column to the "distinct" result set: number of occurences of the entry, similar to uniq -c unix command.
  • STRICT LEFT JOIN is like LEFT JOIN, but generates an error if any key in left table "A" doesn't have exactly one matching key in the right table "B".

Special variables

Variable Name Variable Type Variable Description
a1, a2,..., a{N} string Value of i-th column
b1, b2,..., b{N} string Value of i-th column in join table B
NR integer Line number (1-based)
NF integer Number of fields in line

Aggregate functions and queries

RBQL supports the following aggregate functions, which can also be used with GROUP BY keyword:
COUNT(), MIN(), MAX(), SUM(), AVG(), VARIANCE(), MEDIAN()

Limitations

  • Aggregate function are CASE SENSITIVE and must be CAPITALIZED.
  • It is illegal to use aggregate functions inside Python (or JS) expressions. Although you can use expressions inside aggregate functions. E.g. MAX(float(a1) / 1000) - legal; MAX(a1) / 1000 - illegal.

Examples of RBQL queries

With Python expressions

  • select top 100 a1, int(a2) * 10, len(a4) where a1 == "Buy" order by int(a2)
  • select * order by random.random() - random sort, this is an equivalent of bash command sort -R
  • select top 20 len(a1) / 10, a2 where a2 in ["car", "plane", "boat"] - use Python's "in" to emulate SQL's "in"
  • select len(a1) / 10, a2 where a2 in ["car", "plane", "boat"] limit 20
  • update set a3 = 'US' where a3.find('of America') != -1
  • select * where NR <= 10 - this is an equivalent of bash command "head -n 10", NR is 1-based')
  • select a1, a4 - this is an equivalent of bash command "cut -f 1,4"
  • select * order by int(a2) desc - this is an equivalent of bash command "sort -k2,2 -r -n"
  • select NR, * - enumerate lines, NR is 1-based
  • select * where re.match(".*ab.*", a1) is not None - select entries where first column has "ab" pattern
  • select a1, b1, b2 inner join ./countries.txt on a2 == b1 order by a1, a3 - an example of join query
  • select distinct count len(a1) where a2 != 'US'
  • select MAX(a1), MIN(a1) where a2 != 'US' group by a2, a3

With JavaScript expressions

  • select top 100 a1, a2 * 10, a4.length where a1 == "Buy" order by parseInt(a2)
  • select * order by Math.random() - random sort, this is an equivalent of bash command sort -R
  • select top 20 a1.length / 10, a2 where ["car", "plane", "boat"].indexOf(a2) > -1
  • select a1.length / 10, a2 where ["car", "plane", "boat"].indexOf(a2) > -1 limit 20
  • update set a3 = 'US' where a3.indexOf('of America') != -1
  • select * where NR <= 10 - this is an equivalent of bash command "head -n 10", NR is 1-based')
  • select a1, a4 - this is an equivalent of bash command "cut -f 1,4"
  • select * order by parseInt(a2) desc - this is an equivalent of bash command "sort -k2,2 -r -n"
  • select NR, * - enumerate lines, NR is 1-based
  • select a1, b1, b2 inner join ./countries.txt on a2 == b1 order by a1, a3 - an example of join query
  • select distinct count a1.length where a2 != 'US'
  • select MAX(a1), MIN(a1) where a2 != 'US' group by a2, a3

FAQ

How does RBQL work?

Python module rbql.py parses RBQL query, creates a new python worker module, then imports and executes it.

Explanation of simplified Python version of RBQL algorithm by example.

  1. User enters the following query, which is stored as a string Q:
    SELECT a3, int(a4) + 100, len(a2) WHERE a1 != 'SELL'
  1. RBQL replaces all a{i} substrings in the query string Q with a[{i - 1}] substrings. The result is the following string:
    Q = "SELECT a[2], int(a[3]) + 100, len(a[1]) WHERE a[0] != 'SELL'"
  1. RBQL searches for "SELECT" and "WHERE" keywords in the query string Q, throws the keywords away, and puts everything after these keywords into two variables S - select part and W - where part, so we will get:
    S = "a[2], int(a[3]) + 100, len(a[1])"
    W = "a[0] != 'SELL'"
  1. RBQL has static template script which looks like this:
    for line in sys.stdin:
        a = line.rstrip('\n').split('\t')
        if %%%W_Expression%%%:
            out_fields = [%%%S_Expression%%%]
            print '\t'.join([str(v) for v in out_fields])
  1. RBQL replaces %%%W_Expression%%% with W and %%%S_Expression%%% with S so we get the following script:
    for line in sys.stdin:
        a = line.rstrip('\n').split('\t')
        if a[0] != 'SELL':
            out_fields = [a[2], int(a[3]) + 100, len(a[1])]
            print '\t'.join([str(v) for v in out_fields])
  1. RBQL runs the patched script against user's data file:
    ./tmp_script.py < data.tsv > result.tsv

Result set of the original query (SELECT a3, int(a4) + 100, len(a2) WHERE a1 != 'SELL') is in the "result.tsv" file. It is clear that this simplified version can only work with tab-separated files.

Is this technology reliable?

It should be: RBQL scripts have only 1000 - 2000 lines combined (depending on how you count them) and there are no external dependencies. There is no complex logic, even query parsing functions are very simple. If something goes wrong RBQL will show an error instead of producing incorrect output, also there are currently 5 different warning types.

cli_rbql.py and cli_rbql.js scripts

RBQL comes with cli_rbql.py and cli_rbql.js scripts. Use them as standalone programs to execute RBQL queries from command line.

Usage example:

./cli_rbql.py --query "select a1, a2 order by a1" < input.tsv

To find out more about cli_rbql.py and available options, execute:

./cli_rbql.py -h