examples/tutorial/1_example_setup_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial 1- `AdaptiveMD` basics\n",
    "\n",
    "#### First we cover some basics demonstrating the use of `AdaptiveMD` objects to get you going.\n",
    "\n",
    "We will briefly talk about\n",
    "\n",
    "1. resources\n",
    "2. files\n",
    "3. generators\n",
    "4. how to run a simple trajectory\n",
    "\n",
    "All of these objects from the `adaptivemd` package are used to organize a workflow by associating them with the `adaptivemd.Project` class. We will create a new project called \"*tutorial*\" that is used in this and subsequent tutorial notebooks, so as with any project, be careful about deleting work if revisiting this notebook. All python packages specified in the README must be installed on your local machine, along with MongoDB. The first tutorials can easily be modified to test the use of other resources, so make any necessary adjustments. A `mongodb` port must be visible to this notebook session and the at the time of execution defined `resource`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connecting AdaptiveMD to a MongoDB Instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before getting started, it is essential to connect your AdaptiveMD python session to a running mongo database. You must run the database outside AdaptiveMD, and then configure the connection to it. In the simplest (default) case, it will be located on your local host on the default MongoDB port, 27017, and you don't need to do anything to connect. In a practical scenario, the database will not be located on a local machine. You will need to determine the IP address of the database host, and port number used by the `mongod` instance.\n",
    "\n",
    "Assuming you don't have a firewall issue, you can connect by changing the database network address set in the package. You can always check this looking at the `_db_url` attribute of our MongoDB storage interface. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from adaptivemd import mongodb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mongodb://localhost:27017/'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mongodb.MongoDBStorage._db_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To change the port number, use the `set_port` method of the `MongoDBStorage` interface class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mongodb://localhost:27018/'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mongodb.MongoDBStorage.set_port(27018)\n",
    "mongodb.MongoDBStorage._db_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, reset the host address with `set_host`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mongodb://128.219.191.255:27017/'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mongodb.MongoDBStorage.set_host('128.219.191.255')\n",
    "mongodb.MongoDBStorage._db_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or the whole location at one time with `set_location`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mongodb://localhost:27017'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mongodb.MongoDBStorage.set_location('localhost:27017')\n",
    "mongodb.MongoDBStorage._db_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The database URL used by AdaptiveMD is a class attribute of `MongoDBStorage`, so it is 'global' for a python session. You cannot use multiple databases in a single session. Before loading a `Project`, this must be set correctly, or else you will get a `pymongo` connection error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The `Project`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, let's load the package and import the `Project` class since we want to start a project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from adaptivemd import Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A bit more on connecting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the best way to set the database location is probably with a `Project` class or object. The convenience functions are `set_dbhost`, `set_dbport`, and `set_dblocation` that call the `MongoDBStorage` methods above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mongodb://128.219.186.38:27017/'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Project.set_dbhost('128.219.186.38')\n",
    "mongodb.MongoDBStorage._db_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mongodb://128.219.186.38:27018/'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Project.set_dbport('27018')\n",
    "mongodb.MongoDBStorage._db_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the Project, the whole URL can also be set at one time if you'd like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mongodb://not-correct:dburl/'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Project.set_dburl('mongodb://not-correct:dburl/')\n",
    "mongodb.MongoDBStorage._db_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mongodb://localhost:27017'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Project.set_dblocation('localhost:27017')\n",
    "mongodb.MongoDBStorage._db_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now for the fun stuff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's open a project with a UNIQUE name by instantiating `Project` with a name. This will be the name used in the DB so make sure it is new and not too short. Calling `adaptivemd.Project` with a name to construct an instance will always create a non-existing project, or reopen an exising one. You cannot chose between opening types as you would with a file. This is a precaution to not accidentally delete your project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's see what projects exist already by listing them from the database. Careful not to delete something you want to keep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Project.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'test']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use this to completely remove the tutorial project from the database.\n",
    "Project.delete('tutorial')\n",
    "Project.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if you have trajectories or models saved in a pre-existing folder for a project named tutorial, they are not deleted. They will be overwritten as new data is produced. The new project will iterate through pre-existing trajectory names and overwrite the data as each next tutorial is run. The data must be manually moved if desired, or deleted. Only the MongoDB storage associated with the project has been affected by `Project.delete('tutorial')`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "project = Project('tutorial')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a handle for our project. First thing is to set it up to work on a resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Resource `Configuration`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### What is a configuration? \n",
    "\n",
    "A `Configuration` specifies a shared filesystem and clustere attached to it. This can be your local machine, a regular cluster, or even a group of cluster that can access the same FS (like Titan, Eos and Rhea do). At least one is created for a project, and more may be specified via a configurations.cfg file or object creation in a session. Once created, these should remain static for a project since AdaptiveMD doesn't move data around yet on a (multiple) filesystem(s)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us look at the simplest cases to use a local resource configuration; your laptop or desktop machine for now. No cluster / HPC involved. In a later tutorial, we will consider more ways to set the configuration. `shared_path` is a required field/attribute to tell adaptivemd where to store project data such as simulations and models:\n",
    " - no configuration specified when initializing project --> local config with `shared_path` '$HOME/adaptivemd'\n",
    " - give configuration `dict` to `initialize` method --> `{'shared_path': '$HOME/admd'}`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this object defines the path where all files will be placed, let's get the path to the shared folder. This path must be accessible from all workers on the resource. When using a local resource make sure you have the default folder created on your machine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`initialize` is only run once for a project. When this command is executed, the project is entered in the MongoDB under the name 'tutorial'. The directory for the project is not yet created, as workers manage the files and folders associated with a project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "project.initialize({'shared_path': '$HOME/admd'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'$HOME/admd'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.configurations.one.shared_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `File` Objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from adaptivemd import File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we define a `File` object. Instead of just a string, these are used to represent files anywhere, on the cluster or your local application. There are some subclasses or _extensions_ of `File` that have additional meta information like `Trajectory` or `Frame`. The underlying base object of a `File` is called a `Location`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start with a first PDB file that is located on this machine at a relative path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdb_file = File('file://../files/alanine/alanine.pdb')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`File`, like any complex object in adaptivemd, can have a `.name` attribute that makes them easier to find later. You can either set the `.name` property after creation, or use a little helper method `.named()` to get a one-liner. This function will set `.name` and return itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information about the possibilities to specify filelocation consult the documentation for `File`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdb_file.name = 'initial_pdb'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.load()` at the end is important. It causes the `File` object to load the content of the file, and if you save the `File` object in the database, the actual file is stored with it. This way it can simply be rewritten on the cluster or anywhere else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'alanine.pdb'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pdb_file.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Generator` Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TaskGenerators are instances whose purpose is to create tasks for execution. This is similar to the\n",
    "way Kernels work. A TaskGenerator will generate `Task` objects for you which will be translated into a `TaskDescription` and executed. In simple terms:\n",
    "\n",
    "**The task generator creates the bash scripts for you that run a simulation or run pyemma.**\n",
    "\n",
    "A task generator will be initialized with all parameters needed to make it work, and it will know what files need to be staged for the task to be used. `adaptivemd` relies primarily on two types of generators we will call:\n",
    "\n",
    "1. engine:   for producing simulation data\n",
    "2. modeller: for analyzing simulation data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The `Engine` is used to run trajectories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from adaptivemd.engine.openmm import OpenMMEngine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A task generator will create tasks that workers use to run simulations. Currently, this means a little python script is created that will excute OpenMM. It requires conda to be added to the PATH variable, or at least openmm to be included in the python installation used by the resource. If you set up your resource correctly, then the task should execute automatically via a worker."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's do an example for the OpenMM engine. A small python script is created that makes OpenMM look like a executable. It runs a simulation by providing an initial frame, OpenMM specific system.xml and integrator.xml files, and some additional parameters like the platform name, how often to store simulation frames, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = OpenMMEngine(\n",
    "    pdb_file=pdb_file,\n",
    "    system_file=File('file://../files/alanine/system.xml').load(),\n",
    "    integrator_file=File('file://../files/alanine/integrator.xml').load(),\n",
    "    args='-r --report-interval 1 -p CPU'\n",
    ").named('openmm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now an OpenMMEngine which uses the previously made pdb `File` object in the location defined by its `shared_path`. The same for the OpenMM XML files, along with some args to run using the `CPU` kernel, etc.\n",
    "\n",
    "Last we name the engine `openmm` to find it later, when we reopen the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'openmm'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "engine.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to set the output types we want the engine to generate. We chose a stride of 10 for the `master` trajectory without selection, and save a second trajectory selecting only protein atoms and native stride.\n",
    "\n",
    "Note that the stride and frame number ALWAYS refer to the native steps used in the engine. In our example the engine uses `2fs` time steps. So master stores every `20fs` and protein every `2fs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "engine.add_output_type('master', 'master.dcd', stride=10)\n",
    "engine.add_output_type('protein', 'protein.dcd', stride=1, selection='protein')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The selection must be an `mdtraj` formatted atom selection string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The `PyEMMAAnalysis` modeller "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from adaptivemd.analysis.pyemma import PyEMMAAnalysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object that computes an MSM model from existing trajectories that you pass it. It is initialized with a `.pdb` file that is used to create features between the $c_\\alpha$ atoms. This implementaton requires a PDB but in general this is not necessay. It is specific to my PyEMMAAnalysis show case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "modeller = PyEMMAAnalysis(\n",
    "    engine=engine,\n",
    "    outtype='protein',\n",
    "    features={'add_inverse_distances': {'select_Backbone': None}}\n",
    ").named('pyemma')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again we name it `pyemma` for later reference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We specified which output type from the engine we want to analyse. We chose the protein trajectories since these are faster to load and have better time resolution.\n",
    "\n",
    "The features dict expresses which features to use in the analysis. In this case we will use all inverse distances between backbone c_alpha atoms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add generators to project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next step is to add the generators to the project for later usage. We pick the `.generators` store and just add it. Consider a store that works like a `set()` in python, where additionally when an object is added it is stored in the database. It contains objects only once and is not ordered. Therefore we need a name to find the objects later. Of course you can always iterate over all objects, but the order is not given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#project.generators.add(engine)\n",
    "#project.generators.add(modeller)\n",
    "project.generators.add([engine, modeller])\n",
    "len(project.generators)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, that you cannot add the same engine instance twice (or any stored object to its store). If you create a new but equivalent engine, it will be considered different and hence you can store it again. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.generators.add(engine)\n",
    "len(project.generators)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create one initial trajectory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we are ready to run a first trajectory that we will store as a point of reference in the project. Also it is nice to see how it works in general."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are using a _Worker_ approach. This means simply that someone (in our case the user from inside a script or a notebook) creates a list of tasks to be done and some other instance (the worker) will actually do the work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a `Trajectory` object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we create the parameters for the engine to run the simulation. We use a `Trajectory` object (a special `File` with initial frame and length) as the input. You could of course pass these things separately, but this way, we can actually reference the not yet existing trajectory and do stuff with it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Trajectory should have a unique name, and so there is a project function to do this automatically. It uses numbers and makes sure that this number has not been used yet in the project. The data will be stored in the \"$HOME/admd/projects/`project`.`name`/trajs/`traj`.`name`\" directory we set in the configuration, ie:\n",
    "\n",
    "> $HOME/admd/project/tutorial/trajs/00000000/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Trajectory('alanine.pdb' >> 00000000[0..100])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trajectory = project.new_trajectory(engine['pdb_file'], 100, engine)\n",
    "#trajectory = project.new_trajectory(pdb_file, 100, engine)\n",
    "trajectory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This says, initial is `alanine.pdb` run for 100 frames and is named `xxxxxxxx`. This is the name of a folder in the data directory, where trajectory files will be stored. Multiple atom selections, e.g. protein and all atoms, may be written to create multiple files in this folder. We will refer to these distinct trajectories as the `outtypes` later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Why do we need a trajectory object?\n",
    "\n",
    "You might wonder why a `Trajectory` object is necessary. You could just build a function that will take these parameters and run a simulation. At the end it will return the trajectory object. The same object we created just now.\n",
    "\n",
    "One *main reason* is to use it as a so-called _Promise_ in AdaptiveMD's asynchronous execution framework. The trajectory object we built acts as a _Promise_, so what is that exactly?\n",
    "\n",
    "A _Promise_ is a value (or an object) that represents the result of a function at some point in the future. In our case it represents a trajectory at some point in the future. Normal promises have specific functions do deal with the unknown result, for us this is a little different but the general concept stands. We create an object that represents the specifications of a `Trajectory` and so, regardless of existence (the corresponding data file), we can use the trajectory as if it would exists to build operations on it.\n",
    "\n",
    "We see the *second reason* by considering the object after the promise is fulfilled. We now have an object that can offer a lightweight view on the trajectory data it represents for inspection and sampling. Later we will use it as a convenient way to view analysis results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Trajectory` objects are list-like, get the length:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100\n"
     ]
    }
   ],
   "source": [
    "print(trajectory.length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and since the length is fixed, we know how many frames there are and can access them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "Frame(sandbox:///{}/00000000/[20])\n",
      "False\n",
      "Frame(sandbox:///{}/00000000/[19])\n"
     ]
    }
   ],
   "source": [
    "print(trajectory[20].exists)\n",
    "print(trajectory[20])\n",
    "print(trajectory[19].exists)\n",
    "print(trajectory[19])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`extend` method to elongate the trajectory in an additional task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<adaptivemd.engine.engine.TrajectoryExtensionTask object at 0x112ab5490>\n"
     ]
    }
   ],
   "source": [
    "print(trajectory.extend(100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`run` method gives us a `task` that will do an MD simulation and create the trajectory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<adaptivemd.engine.engine.TrajectoryGenerationTask object at 0x112ab5a10>\n"
     ]
    }
   ],
   "source": [
    "print(trajectory.run())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can ask to extend it, we can save it. We can reference specific frames in it before running a simulation. You could even build a whole set of related simulations this way without running a single frame. This is pretty powerful especially in the context of running asynchronous simulations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Create a `Task` object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we want that this trajectory actually exists so we have to make it. This requires a `Task` object that _knows_ how to describe the execution of a simulation. Since `Task` objects are very flexible and can be complex, there are helper functions (i.e. factories) to create these in an easy manner like the ones we created before."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the trajectory (which uses its engine) to call `.run()` and save the returned task object to directly work with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "task = trajectory.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it, just take a trajectory description and turn it into a task that contains the shell commands and needed files, etc. Use the property `trajectory.exists` so see whether the trajectory object is associated with any data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trajectory.exists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submit the task to the queue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we need to add this task to the things we want to be done. This is easy and only requires saving the task to the project. This is done to the `project.tasks` bundle and once it has been stored it can be picked up by any worker to execute it.\n",
    "\n",
    "Note that you should be able to submit a trajectory like this, however in practical situations it is likely some additional operations are required in the pre- and post- tasks (outside scope of this tutorial), so they will usually be converted to tasks prior to queueing in project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#FIXME#project.queue(trajectory)\n",
    "# shortcut for project.tasks.add(task)\n",
    "project.queue(task)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(project.tasks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is all we can do from here. To execute the tasks you need to create a worker using the `adaptivemdworker` command from the shell:\n",
    "\n",
    "```bash\n",
    "adaptivemdworker tutorial --verbose\n",
    "```\n",
    "\n",
    "For the simple setup in this tutorial, you can just navigate to the directory as follows if it's not in your PATH already.\n",
    "\n",
    "```bash\n",
    "cd home/of/adaptivemd/scripts/\n",
    "```\n",
    "\n",
    "The worker is responsible for managing the project filestructure, so when both the worker is running and `project` entries and structure are changed in the database, the directory and subdirectories of \"$shared_path/project/tutorial\" are created and modified. The project subfolders \"trajs\" and \"models\" are populated with trajectories and models of the corresponding names as workers complete tasks that are entered into the database.\n",
    "\n",
    "If you have a database set up remotely, you will need to tell the worker where to look for this connection just as with the AdaptiveMD application. There are 3 options you can use for this:\n",
    "1- \"-d\" to set the database host IP\n",
    "2- \"-p\" to set the database port on its host\n",
    "3- \"-l\" to set the full database location, optionally with the URL prefix \"mongodb://\"\n",
    "\n",
    "i.e.\n",
    "Host IP\n",
    "```bash\n",
    "adaptivemdworker tutorial -d 8.8.8.8\n",
    "```\n",
    "\n",
    "Host port number\n",
    "```bash\n",
    "adaptivemdworker tutorial -p 27018\n",
    "```\n",
    "\n",
    "Full location\n",
    "```bash\n",
    "adaptivemdworker tutorial -l 8.8.8.8:27018\n",
    "```\n",
    "or\n",
    "```bash\n",
    "adaptivemdworker tutorial -l mongodb://8.8.8.8:27018/\n",
    "```\n",
    "or\n",
    "```bash\n",
    "adaptivemdworker tutorial -d 8.8.8.8 -p 27018\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'dummy', u'created']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task.state\n",
    "project.tasks.all.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now there are data files & folders associated with the trajectory\n",
    "project.wait_until(task.is_done)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'success'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task.state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are done for now, its also good practice to relieve your workers (and save yourself some compute time charges on HPC resources!). You don't have to, even if you're closing the project's database connection. They are associated with the project and will accept tasks at any point that are entered in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use the 'one' method inherited from bundle to see available methods\n",
    "# for the worker type, such as 'execute'\n",
    "#project.workers.one.execute('shutdown')\n",
    "\n",
    "# but use 'all' method in practice to apply across all members of\n",
    "# the workers bundle in the typical case, where you have many workers\n",
    "project.workers.all.execute('shutdown')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final `project.close()` will close the DB connection. The daemon outside the notebook would be closed separately. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "project.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}