The database

The database

Database operations are fully integrated in the Opa language, which increases code re-usability, simplifies code coverage tests, and permits both automated optimizations and automated safety, security and sanity checks.

In this chapter, we describe the constructions of Opa dedicated to storage of data and manipulation of stored data.

Declaring a database

The core notion of the database is that of a path. A database path describes a position in the database and it is through a path that you can read, write and remove data.

A database is named and consists of a set of declared paths. Opa handles the popular MongoDB database backend.

The following piece of code declares a database named my_db and defines :

• 3 paths containing basic values (/i, /f, /s)
• 1 path containing a record (/r)
• 1 path containing a list of strings
• 1 path to a map from integers (intmap) to values of type stored.
• 1 path to a set of stored records, where the primary key of the declared set is x.
• 1 path to a set of more complex records where the primary key of the set is id. This most complex records contain embedded maps and lists.

    type stored = {int x, int y, string v, list(string) lst}

    // where _db_options_ allows to select the database backend, see below
    database my_db {
      int    /basic/i
      float  /basic/f
      string /basic/s
      stored /r
      intmap(stored) /imap
      stored /set[{x}]
      {int id, stored s, intmap(stored) imap, stringmap(stored) smap} /complex[{id}]
    }

Initializing database

Opa applications can use several databases with several different backends. For each database a command line family is generated. When running your applications you can specify, via command line arguments, options for each declared database.

Both backends have common command lines options :

• --db-local:dbname [/path/to/db]: Use a local database, stored at the specified location in the file-system.
• --db-remote:dbname host(s): Use a remote database accessible at a given remote location.

Note that when you use for the first time the local database options with MongoDb, the Opa application will download, install, and launch a single instance of MongoDb database.

Note also that you can define database authentication parameters for each database with the syntax --db-remote:dbname username:password@hostname:port.

Database reads/writes

One can read from the /my_db/basic/i path with:

x = /my_db/basic/i

and write the contents of said path:

/my_db/basic/i <- x

Note that the database is structured. Therefore, if you access path /my_db/basic you will obtain a value of type { int i, float f, string s }.

Conversely, you could have defined the path /my_db/basic as:

...
{ int i, float f, string s } /basic
...

and then still access paths /my_db/basic/i, /my_db/basic/f or /my_db/basic/s directly.

Default values

Each path has a default value. Whenever you attempt to read a value that does not exist (was never initialized or was removed), the default value is returned.

While Opa can generally define a default value automatically, it is often a good idea to define an application-specific default value:

...
string /basic/s = "default string"
...

If you do not define a default value, Opa uses the following strategy:

• the default value for an integer (int) is 0;
• the default value for a floating-point number (float) is 0.0;
• the default value for a string is "";
• the default value for a record is the record of default values;
• the default value for a sum type is the case which looks most like an empty case (e.g. {none} for option, {nil} for list, etc.)

Sub-paths

Sub-paths are paths relative to some location in the database. If you think about paths as akin to absolute paths in the file-system, then sub-paths are a bit like relative paths.

There are several ways to build sub-paths, that can be further composed by by separating them with dots (., as opposed to / used for paths).

• A record field (<ident>) accessing a record field with the given name,
• An indexed_expression ([<expr>]) accessing elements of a map with constaints given by <expr> and
• A hole expression ([_]) acessing elements of a list.

We'll talk more about sub-path in the following chapters.

Querying

Opa 0.9.0 introduced database sets and a powerful way to access a subset of database collections.

A query is composed from the following operators:

• == expr: equals expr
• != expr: not equals expr
• < expr: lesser than expr
• <= expr: lesser than or equals expr
• > expr: greater than expr
• >= expr: greater than or equals expr
• in expr: "belongs to" expr, where expr is a list
• q1 or q2: satisfy query q1 or q2
• q1 and q2: satisfy queries q1 and q2
• not q: does not satisfy q
• {f1 q1, f2 q2, ...}: the database field f1 satisfies q1, field f2 satisfies q2 etc.

Furthermore you can specify some querying options:

• skip n: where expr should be an expression of type int, skip the first n results
• limit n: where expr should be an expression of type int, returns a maximum of n results
• order fld (, fld)+: where fld specify an order. fld can be a single identifier or a list of identifiers specifying the fields on which the ordering should be based. Identifiers can optionally be prefixed with + or - to specify, respectively, ascending or descending order. Finally it's possible to choose the order dynamically with <ident>=<expr> where <expr> should be of type {up} or {down}.

Querying database sets

k = {x : 9}
stored x = /dbname/set[k] // k should be type of set keys, returns a unique value
stored x = /dbname/set[x == 10] // Returns a unique value because {x} is the primary key of the set

dbset(stored, _) x = /dbname/set[y == 10] // Returns a database set because y is not a primary key

dbset(stored, _) x = /dbname/set[x > 10, y > 0, v in ["foo", "bar"]]

dbset(stored, _) x = /dbname/set[x > 10, y > 0, v in ["foo", "bar"]; skip 10; limit 15; order +x, -y]

it = DbSet.iterator(x); // then use Iter module.

Querying database maps

// Access to a unique value (like in Opa <0.9.0 (S3))
stored x = /dbname/imap[9]

// Access to a submap where keys are smaller than 9 and greater than 4
intmap(stored) submap = /dbname/imap[< 9 and > 4]

Sub-queries

We can also use sub-paths to perform sub-queries on elements at a given path.

With indexed-expressions you can query on values inside maps.

// Querying using expression fields.
// This query returns all elements in the database set at path
// `/dbname/complex` where
// - The field `x` of the `stored` record on field `s` of database set
//   elements is smaller that 100
// - and the intmap `imap` contains an elements associated to the key `10`
//   where the field `v` is equal to `"value"`
// - and the stringmap `smap` contains a bindings with the key `"key"`
dbset(_, _) x = /dbname/complex[s.x < 100 and imap[10].v == "value" and smap["key"] exists]

With hole expressions you can query elements within a list.

// Querying using hole expressions.
// This query returns all elements for which the stringmap `smap` contains a
// value associated to the `"key"` and the list on the field `lst` contains a
// value equal to "value".
dbset(_, _) x = /dbname/complex[smap["key"].lst[_] == "value"]

Updating

Previously we saw how to overwrite a path, like that :

/path/to/data <- x

which assigns to path /path/to/data the value of x. This used to be the only way to write to a path, but since Opa 0.9.0 there are more ways to update them.

Int updating

// Overwrite
/my_db/basic/i <- 42

// Increment
/my_db/basic/i++
/dbname/i += 10

// Decrement
/my_db/basic/i--
/my_db/basic/i -= 10

// Sure we can go across records
/my_db/r/x++

Record updating

// Overwrite an entire record
x = {x: 1, y: 2, v: "Hello, world", lst: []}
/my_db/r <- x

// Update a subset of record fields
/my_db/r <- {x++, y--}

/my_db/r <- {x++, v : "I just want to update z and incr x"}

List updating

// Overwrite an entire list
/my_db/l <- ["Hello", ",", "world", "."]

// Removes first element of a list
/my_db/l pop

// Removes last element of a list
/my_db/l shift

// Append an element
/my_db/l <+ "element"

// Append several elements
/my_db/l <++ ["I", "love", "Opa"]

// Remove an element
/my_db/l <--* "element"

// Remove several elements
/my_db/l <-- ["I", "hate", "Opa"]

Set/Map updating

The values stored inside database sets and maps can be updated as above. The difference is how we access the elements (more details in the querying section). Furthermore updates can operates on several paths.

//Increment the field y of the record stored at position 9 of imap
/dbname/imap[9] <- {y++}

//Increment fields y of records stored with key smaller than 9
/dbname/imap[< 9] <- {y++}

//Increment the field y of record where the field x is equals 9
/dbname/set[x == 9] <- {y++}

//Increment the field y of all records where the field x is smaller than 9
/dbname/set[x < 9] <- {y++}

Removing

// Remove the counter value
Db.remove(@/dbname/counter)

// Remove the item that has the primary key 9 in the set
Db.remove(@/dbname/set[id == 9])

Note the use of @ in @/dbname/path: it returns a reference to the path, where /dbname/path would have returned the value at this path.

Updating sub-paths

With sub-paths it is possible to update some values deeper in the database structure, relative to the point of update.

For instance, with indexed-expressions it's possible to write:

// Updating using expression fields.
// Update elements which match the query `...` by:
// - Incrementing field `x` of the `stored` record on field `s` of the database set
// - and add 2 to the field `x` and substract 2 from the field `y` of the
//   element associated to the `"key"` in the stringmap `smap`
/dbname/complex[...] <- {s.x++, smap["key"].{x += 2, y -= 2}}

With hole expressions it is possible to update the first element of a list matched by the corresponding hole in the query.

// Updating using hole fields.
// Write "another" into the first element of the list `lst` in the map `smap`
// at key "key".
/dbname/complex[smap["key"].lst[_] == "value"] <- {smap["key"].lst[_] : "another"}

Projection

We saw how we can query database sets and maps to obtain a sub-set of their values. But often we do not need all the information stored there but only a selected few fields and fetching complete information from the database would be inefficient.

We already saw how we can access a subpath of a given path:

// Access to /my_db/r
stored x = /my_db/r
// Access to the /x sub-path of /my_db/r
int x = /my_db/r/x

If we want to access to both subpath /x and /y of path /my_db/r we can write something like that:

{int x, int y} xy = {x : /my_db/r/x, y : /my_db/r/y}

But there is a more concise syntax to achieve the same:

// This path access means 'select fields x and y in path /my_db/r'
{int x, int y} x = /my_db/r.{x, y}

Similarly, you can use these kinds of projections on database sets and maps. Some examples follow:

// Access of /v sub-path of a set access with primary key
string result = /dbname/set[x == 10]/v

// Access of both sub-path /x and /v of a set access with primary key
{int x, string v} result = /dbname/set[x == 10].{x, v}

// Access of /v sub-paths of a set access
dbset(string, _) results = /dbname/set[x > 10, y > 0, v in ["foo", "bar"]]/v

// Access of two sub-paths /x and /v of a set access
dbset({int x, string v}, _) results = /dbname/set[x > 10, y > 0, v in ["foo", "bar"]].{x, v}

You can also use sub-paths to project selected data.

dbset({{int x, int y} s, intmap(stored) imap}) results = /dbname/complex[...].{s.{x, y}, imap}

dbset({{int x, int y} s, stored imap}) results = /dbname/complex[...].{s.{x, y}, imap["key"]}

Manipulating paths

Most operations deal with data, as described above. We will now describe operations that deal with meta-data, that is the paths themselves.

A path written with notation !/path/to/data is called a value path. Value paths represent a snapshot of the data stored at that path. Should you write a new value at this path at a later stage, this will not affect the data or meta-data of such snapshots.

// Getting a value path. i.e. a read only path
!/path/to/data

Should you need to make reference to the latest version of data and meta-data, you will need to use a reference path, as obtained with the following notation:

// Getting a reference path. i.e. a r/w path
@/path/to/data

A path value is specific to its own database backend, indeed the type of a path contains 2 type variables; the first one for the type of data stored in the path and the second one for specifying the database backends used for this path.

For instance if my_db is declared as a MongoDb database, here is type of its paths :

Db.val_path(string, DbMongo.engine) !/my_db/s
Db.ref_path(string, DbMongo.engine) @/my_db/s

Generic functions to manipulate such values are available in the Db module (imported by default).

Note, that the same mechanism is used for dbset('data, 'engine)

Currently, the MongoDb driver does not verify if the compiled application is compatible with a database schema. That means that after changing data used in the application the developer herself needs to take care of migrating the data.