Introduction to the MDR

The MDR, or Metadata Repository, is a software tool that supports a standardized data description.

It uses so-called Namespaces to hold sets of descriptions. Multiple namespaces are possible. This allows, for example, versioned data descriptions, where multiple versions are available concurrently. It also permits the use of different data models in different projects. Namespaces must have unique names.

This repository bundles the MDR components together with Samply.auth, to make it easy to install and run your own MDR.

Components

The MDR has 4 components:

1. Database. This stores the Namespaces, users, and settings.
2. REST service. This provides an API that can be used to query the MDR Namespaces. This is what the Sample Locator GUI connects to.
3. UI. This has a regular user interface, that can be used to edit Namespaces, plus an admin interface, which can be used for configuration, users and permissions.
4. Authentification. Here, we use Samply.auth.

URLs

The MDR and Samply.auth are configured via their own admin GUIs.

If you are setting up on a server with a publicly-accessible URL, you could make the MDR and Samply.auth ports (8087 and 8086 respectively) accessible to the outside world.

Alternatively (and probably safer), you could set up a "fake" URL that is only known to your server. In order to use this, your server will need its own desktop. You can then access it via a remote desktop service, such as x2go.

If you are simply working with your own local computer (e.g. laptop), then this is definitely the best way to go.

We want to simulate the situation where each component has its own URL. To do this, you need to add the following line to the "hosts" file on your machine:

    127.0.0.1       mdr-samplyauth

Under Windows, you will find this file here:

    C:\Windows\System32\drivers\etc\hosts

Under Linux, look in:

    /etc/hosts.

You will also need to flush the DNS-cache if you are using Linux.

By making these changes, you ensure that the two web pages get redirected to 'localhost'. In the docker-compose.yml file, the relevant containers are so named that these names are also used internal to the Docker network. This means that the servers are known by the same names both inside the Docker network and outside of it. That's the trick needed to make callbacks work, which are important for authentication.

Note that you should try to avoid using proxies or your institutional network when doing this kind of thing. Eduroam works pretty well.

Starting the MDR

In this directory, run the command:

    docker-compose up -d

This will start all 4 of the services described above.

To stop the MDR:

    docker-compose down

Setting up Samply.auth

You need to log in to the MDR in order to be able to do anything useful. This can be done via Samply.auth.

A local Samply.auth server can be used. Once you have set it up, you should change the admin password. This can be set with the TOMCAT_USERNAME and TOMCAT_PASSWORD environment variables.

You then need to set up a Client in Samply.auth. I used a "Client type" undefined, and it worked fine, but I noticed that in all of the departmental Samply.auth servers, the client type was set to "Metadata repository". So it probably doesn't do any harm to use that setting. Use the following settings for the client in the Samply.auth admin GUI (http://mdr-samplyauth:8086/admin):

    Client Name: mdr_ui
    Client Description: mdr_ui
    Redirect URLs: http://mdr-samplyauth:8087
    Client ID: mdr_ui (overwrite the default value)
    Client Secret: You can't change this. You will need to copy it into the docker-compose.yml file.
    Client type: Metadata Repository

If you are using Docker, make sure that Samply.auth shares the same network as the MDR and MDR-UI.

You will need to create a user in Samply.auth to be used when logging in to MDR.

You will also need a public key for the docker-compose file. You can find this at:

    http://mdr-samplyauth:8086/oauth2/certs
    
    (base64DerFormat)

Setting up MDR

The MDR and MDR-UI have environment variables that allow you to point them at Samply.auth. The client name and is the name that you used when you set up the client in Samply.auth. The client secret is generated by Samply.auth, and you can find it if you look at the "Details" for the client. You will also need a public key for Samply.auth. You can find this by typing in the URL <Samply.auth base>/oauth2/certs. E.g. for this docker-compose file: http://mdr-samplyauth:8086/oauth2/certs. The relevant key is "base64DerFormat".

After your first successful login to the MDR, a user will be created in the MDR database, which can be used for setting permissions.

By default, this user will have very limited capabilities, it will only be able to see which Namespaces are present, and browse the contents of the Namespaces.

If you want to be able to edit, you need enhanced permissions, which only the MDR admin can give you.

If you have installed your own local MDR (e.g. http://mdr-ui.site.de:8087/admin/), then the admin login will, by default, be "admin", with a password "adminpass". To find out the password on an existing system, you will need to log in to the server where it is running and seek the file tomcat-user.xml. Search for tomcat-users. It should look something like this:

    <?xml version='1.0' encoding='utf-8'?>
    
    <tomcat-users>
        <role rolename="mdr-admin"/>
        <user username="justus" password="kkkwwweeeeoooo9999" roles="mdr-admin"/>
    </tomcat-users>

Here you can easily pick out the admin name and password.

When you have logged in as admin, click on the "Users" menu. Assuming you have already logged in as a regular user, you should see yourself on this page, with various permissions. By clicking on the little arrow on the right, you can choose which permissions to activate. You should activate all of them.

Editing Namespaces with the MDR

Once the permissions have been set properly, you can kill the admin window and login as a normal user. Note, it sometimes takes up to 6 login attempts before you can log into the MDR-UI.

After login, you should now have a menu "New".

To import a Namespace as an XML file, you first need to create a new Namespace with the menu option "New Namespace". For compatibility with the rest of the Sample Locator infrastructure, I strongly advise you to call the Namespace "mdr16". Then under the menu option "Select namespace", select the Namespace you just created. Finally, from the "New" menu, run the "Import file" wizard. Make sure you tick "Keep identifiers and versions".

Note that the MDR is very picky about dataspace naming. Stick to standard ASCII alphanumeric characters.

In the "data" directory are several example data models that you can import into a new dataspace and build upon, if you wish. All of them are based on the GBA data model, with small variations.

If you want to clone a Namespace so that you can extend it, you need to first export it and then import it into a nwe Namespace that you have created for the purpose.

Note that you can't change a namespace's name once it exists, and you can't delete it either. So think carefully before you create a new one.

Namespaces are hierarchical constructs, composed of Dataelements, Dataelement groups, and Records. The mdr16 Namespace uses exclusively Dataelements and Dataelement groups. At the top level is a list of Dataelement groups, which define Samples, Donors, etc. These contain Dataelements, which are typed attributes.

To add a new element anywhere in the hierarchy, you first use the "New" menu to create it. It will initially be placed at the top level of the hierarchy. You will be led through a wizard, which will allow you to name the element and set its characteristics. You will generally not want to add any slots.

You can then add the new element to any Dataelement group, by clicking on the group, and editing it via the button in the top right corner. You will be taken into a wizard. One of the steps will be called "Assign members and subgroups". Here, you can select the Namespace where you added the new element. You may need to scroll down to the element. You can then drag the element and drop it into the box on the left. You can also reorder the elements here. Now finish off the edit wizard. If you look at the Namespace where you created the new element, you will see that it is gone, but if you look at the Dataelement group that you just edited, you will see that it has been inserted there.

To export a namespace (e.g. if you want to save your edits), click "New" and select the namespace. Then click the View menu, select the same namespace (on the left), and then click on the button on the top right with the three lines. Select "Export this element". An "Export" Menu should appear. Click it. You will then be presented with a window, where you can either do a full export or a regular export. Choose the latter option. You will need to specify file name and path.

Connecting the Sample Locator to the MDR

The Sample Locator GUI needs to be given the URL of the MDR in order to be able to display properly.

If you use one of the standard MDRs, such as https://mdr.test.bbmri.de/v3/api/mdr (the default value), the Sample Locator will happily connect to it and display normally.

However, if you set it to point to a local MDR instance (http://mdr-samplyauth:8080/v3/api/mdr) via the MDR_API_URL variable in docker-compose.yml, then you never get past the "looking for MDR" phase when loading the Sample Locator GUI. Most of the GUI simply will not load and you won't be able to run detailled searches.

This turns out to be a CORS (cross site scripting) issue with mdr-samplyauth:8080, the MDR REST server.

To fix this, install the CORS Everywhere (Cors E) plugin from spenibus into your browser, and click the icon to make it green. Then reload the Sample Locator the page. Once you have done that, the page will load properly and you will be able to run searches as normal.

This is not a very clean solution, but for a local test installation, it is OK. Don't forget to turn CORS back on once you are finished.

Namespaces and the Sample Locator source code

The Sample Locator source code is split into two repositories for server and client:

https://github.com/samply/share-broker-rest https://github.com/samply/sample-locator

If you do a recursive search for "urn:mdr16:dataelement", you will get several hits in both repositories. What this means is that the URNs generated semi-accidentally by the MDR software for mdr16 have been hard-coded into the Locator source code.

So if you want to use your own Namespace in the Sample Locator, you have two choices:

1. Create the Namespace from scratch, e.g. mdr16ejprd, and then replace all of the references to the mdr16 Namespace in the sources with references to the new namespace. Note that the numbers and versions of the elements in your new Namespace will not generally correspond to those in the mdr16 Namespace, so you will need to find out how all of the new elements map onto the old elements, and then do a recursive search and replace over both repositories and then rebuild them.
2. Create a namespace called mdr16, import the original mdr16, and then add new stuff to it. This is not completely trivial, because if you add a new attribute to an existing data element group, the version number will automatically be incremented, rendering it invisible to the Sample Locator. The MDR does not let you manually reset version numbers. To get around this, you will need to export the Namespace into a file, and edit the file to set all versions back to 1. Then import it back into the MDR. Of course, because you can't delete namespaces, you can't get rid of the old mdr16 namespace and import the new one. In my case, I created a new Docker volume for the MDR database, and imported the modified Namespace into a freshly created empty mdr16 Namespace. A hacky approach, but it worked.

I chose option 2, because I didn't want to make deep changes to the source repositories that might conflict with future edits by other people.

Securing your data

The process of setting up users, importing data models and editing them can be time-consuming, so you will probably want to make backup copies of the database where all of this information is stored. A script is packaged with this reporitory for just that purpose. Run:

    sh db_dump.sh

This will create the directory "mdr-db" (if not already present) and dump the database into a timestamped file in the directory.

If you ever need to restore this data, e.g. to revert to a previous iteration of your data model, run:

    sh db_restore.sh mdr-db/<filename>.sql

Note, this will irrevocably overwrite whatever you currently have in the database, so use with care.