05_09_23

Language Agnostic API

This page contains coarse grain specifications for a MIVOT implementation on both server and client sides. The given examples can easily be translated into any language although with an obvious Python flavor.

The purpose of these specifications is in between a common API and an algorithm definition.

Client Side

In this section:

• we suppose that the input is always a VOTable, annotated or not.
• If it is annotated, the models mapped are not known a-priori

The purpose of the MIVOT interpreter is to provide different levels of model views on the data.

• SPEC1 : the MIVOT processing package must not alter the usual work of the VOTable parser. It is a functionality that can be triggered on demand.

General specifications

• SPEC2: if the VOtable is not annotated, all the usual client functionalities must remain available.
• SPEC3: if the VOtable is annotated but the client decide to ignore it, all the usual functionalities must remain available.
• SPEC4: if the VOtable is annotated, the client must be able to tell the users which models are mapped.
• Example:

votable = vo_table("myvotable.xml")
print(votable.get_first_resource().get_mapped_models())
<Measure, Coords, PhotDM, Mango>     

Keep the streaming readout working

• SPEC5: The MIVOT features must work with clients streaming table data.

• Example:

  • non Mivot data streaming

    while row = rows.next()
      flatview = row.data
    

  • Mivot data streaming

    while row = rows.next()
       modelview = model_view(row.data)
    

• SPEC6: If the mapped objects is spread out over several tables, the client must be able to stream the primary table even if this requires to operate other iterators searching the joined data.

• Example:

  • Mivot data streaming

    while row = rows.next()
       modelview = model_view(row.data)
       joined_data = modelview.get_joined_data(which_data)
    

Five layers of model views

In order have code both model agnostic and understanding the most popular models, we define 5 levels of models views

Level1: parser access

• SPEC7: The model layer must provide access to the parser elements describing a model view (e.g. DOM or XML). It must also provide access to sub-element of the model views.
• Example (à la xml):

while row = rows.next()
   parsermodelview = parser_model_view(row.data)
   subelement = parsermodelview.xpath("./ATTRIBUTE[dmrole='model:radial_velocity'")

Level2: parser wrapper

• SPEC8: The model layer must provide accessors facilitating the use of the parser elements required in SPEC7.
• Example (à la xml):

while row = rows.next()
   parsermodelview = parser_model_view(row.data)
   subelement = parsermodelview.get_attribute_by_role('model:radial_velocity')

Level3: model agnostic objects

• SPEC9: The model layer must be able to dynamically generate (as far as the language allows) mock classes matching the mapped objects.
• Example: let's suppose the Mango model has a Position class with 2 attributes (latitude and longitude):

while row = rows.next()
   modelview = model_view(row.data)
   sky_position = modelview.get_instance("mango:Position)
   ra = sky_position.longitude
   dec = sky_position.latitude

Level4: model specific objects

• SPEC10: The model layer must include classes based on the most popular models. This classes might contain some specific logic making easier the discovery of the content of the mapped data.

• Example: let's suppose our data is mapped on Mango model:

while row = rows.next()
   modelview = model_view(row.data)
   mango_view = modelview.get_mango_view("mango:Position)
   for mango_parameter in mango_view.parameters
       if mango_parameter.is_position() and mango_parameter.get_error() < 5
           mango_parameter.plot_in_aladin()

Level5: Wrapper to science objects

• SPEC11: If the host framework contains built-in science classes, the model layer must be able to instantiate such classes from the mapping in a transparent way if possible. The binding between the mapping and built-in classes should likely be based on well known component models such as Meas/Coords.

• Example: let's suppose our framework has a built-in class named sky_coord:

while row = rows.next()
   modelview = model_view(row.data)
   sky_coord = modelview.get_sky_coord()
   ra = sky_coord.longitude
   dec = sky_coord.latitude

Server Side

The process of deploying model annotations is basically 3 folds

1. Rules telling how tables must be mapped
2. A deep API capable of generating MIVOT elements from indications read in 1)
3. An API stacking elements provided by 2) a query responses

Querying annotated data.

In the following document, we suppose that the query responses are annotated on demand with the FORMAT parameter. Example:

• Getting MANGO annotations : query?format=application/mango&query=SELECT….
• Getting CUBE annotations : query?format=application/cube&query=SELECT….
• Getting no annotations : query?format=text/xml;votable&query=SELECT….

That way a service can serve data annotated with different models. We do not talk here about the way o expose this capability.

Mapping rules

The most convenient place to store mapping rules is the TAP schema. As a same service is possibly able to generate annotations related to different models, we suggest setting one resource per model. This is a table named model_mivot where model is the model name in lower case by convention. As one of the main MIVOT use cases is the possibility to group quantities we need a second table for this purpose: model_mivot_associations. The way this second table is consumed depends on the way the model is built. The point is that it joins mapped classes in an unambiguous way. In any case, this is not mandatory.

TAP_SCHEMA: 
    - model_mivot
    - model_mivot_associations

Table model_mivot

This table contains one row for each table column that can be mapped. The set of columns to be mapped in one model class are grouped in our example but this is not necessary. The same table column can be mapped in different model instances, and thus appears more than once in model_mivot table.

This table has the following columns

• instance_id: (MAND) Identifier of each mapped instance
• table: (MAND) qualified name of the table concerned by the current mapping rule.
• column: (MAND) name of the mapped column
• dmtype: (MAND) model class being mapped
• dmrole: (MAND) role the column plays in the current model class
• dmerror: (OPT) identifier of the mapped error attached to the current instance. This identifier must be resolvable in the current table.
• frame: (OPT) definition of the coordinate frame to be associated with the current instance. The field might contain encoded values such as EPOCH or EQUINOX.
• ucd: (OPT) UCD attached to the current instance
• vocab: (OPT) Vocabulary definition that might be associated with the current instance.

instance _id	table	column	dmtype	dmrole	dmerror	frame	ucd	vocab
__main_pos	epic_src	sc_ra	meas:LonLatPos	lon	_mainpos_error	FK5(eq=J2000, ep=2015)	pos.main	#POS
__main_pos	epic_src	sc_dec	meas:LonLatPos	lat	_mainpos_error	FK5(eq=J2000, ep=2015)	pos.main	#POS
_mainpos_error	epic_src	sc_err_min	meas:Ellipse	min
_mainpos_error	epic_src	sc_err_major	meas:Ellipse	maj
_mainpos_error	epic_src	sc_err_angle	meas:Ellipse	angle

Table model_mivot_associations

This table contains the links between various objects declared in model_mivot. The way to process associations is model dependent. It is implemented at the level of annoter engine.

This table has the following columns

• source_instance_id: (MAND) id of the instance hosting the association
• target_instance_id: (MAND) id of the associated instance.

API for setting up the TAP_SCHEMA

The functions below only manage the TAP_SCHEMA tables; any other action such as modifying the declared capabilities are operated at a higher level.

Environment checking

mivot_check_env()

Checks that the different paths given in the environment do exist and are writable

• example: VODML_PATH, SNIPPET_CACHE ….
• Raise an exception if something is wrong

Create mapping tables

mivot_schema_init(model, with_association=False)

Create the table TAP_SCHEMA.model_mivot

• If with_association is set as True, the table TAP_SCHEMA.model_mivot_associations is also created.
• An exception is risen if the table already exists

Drop mapping tables

mivot_drop_schema(model)

• Drop the table TAP_SCHEMA.model_mivot
• Drop TAP_SCHEMA.model_mivot_associations if it exists.

Set rules for the mapping of a particular class.

instance_id = mivot_add_mapped_class(
                    model,
                    table, 
                    Dmtype,
                    {column: {dmrole, frame=None}, …}, 
                    ucd=None, 
                    vocab=None,
                    instance _id=None)

• Add a mapping to a model class for a particular table.
• This class uses the VODML file to check the consistency of the parameters
• Return an auto-generated instance_id if not given as a parameter
• Rise an exception if
  • the instance_id already exist
  • dmtype does not exist in the model
  • Some role does not exist in the model class
  • If some frame does not match the model sub-setting rules

Drop one mapped class

mivot_drop_mapped_class(instance_id)

Drop a mapped class with its associations.

• This class uses the VODML file to check
• Rise an exception if
  • Instance_id does not exist in the table

Add one error object to a mapped class

mivot_add_error_to_mapped_class(
                    instance_id, 
                    error_id)

Add an error object to the mapped class.

• The error mapping must have been declared before with mivot_add_mapped_class
• Rise an exception if
  • instance_id has already an error
  • error_id does not exist in the table
  • The error type breaks the model rules

Drop the error object of a mapped class

mivot_drop_error_from_mapped_class(
                    instance_id)

Drop the error from the class instance_id if it exists

Add an association

mivot_add_association(
                    model,
                    target_instance_id, 
                    ass_instance_id)

Add ass_instance_id to the objects associated to target_instance_id in the table model_mivot_associations.

• Rise an exception if
  • One of the given IDs does not exist in model_mivot
  • Both IDs are the same
  • The same association already exist
  • The reverse association already exist

Drop an association

mivot_drop_association(
      model,
      target_instance_id
)

• Drop all associations with target_instance_id
• Rise an exception if
  • target_instance_id does not exist in model_mivot