Creating a Plugin

Gregory Sharov edited this page Oct 27, 2018 · 8 revisions
Scipion Logo

A Plugin for Scipion is a Python module that have some extra requirements. Each Plugin contains classes for protocols, viewers, wizards, etc…​allowing to use the functionality of a given EM package within the Scipion framework. Prior to version XXX, the plugins code was not completely isolated, it was under the scipion/pyworkflow/em/packages folder and hosted in the same central repository.

For the release XXX ("the pluginization"), we worked hard to make the entire system more de-centralized and more standard. For that we re-factored the plugins to make them standard Python modules and host their code in independent repositories and with the possibility different release cycles.

This document is organized in two main parts: (1) Explanation of the structure of a plugin as a Python module and the required submodules and functionality for Scipion. (2) Necessary files structure and the process for distributing a Plugin in the Python de-facto repository PyPI.

Naming Conventions

Similar to style guides, conventions are about consistency. Consistency is crucial for a project such as Scipion and its distributed nature. Despite the final decision is up to the developer of a package, we STRONGLY recommend the following conventions:

  • Repository name: scipion-em-packagename

  • Python package name: scipion-em-packagename

  • Python module with package code: packagename

This means that it will be a folder named scipion-em-package with the structure of a standard Python package (explained below in the second part) and it will contains a folder package that is a Python module with functions and classes required by Scipion (explained below).

Plugin Overview

Files structure

The files and folders structure follows the convention established for Python modules. For example, the Appion plugin has the following structure:

$ cd ~/work/development/scipion-em-plugins/scipion-em-relion
$ tree
.
├── LICENSE
├── .gitignore
├── CHANGES.txt
├── README.rst
├── MANIFEST.in
├── setup.py
└── relion
    ├── __init__.py
    ├── bibtex.py
    ├── constants.py
    ├── convert
    │   ├── __init__.py
    │   ├── convert.py
    │   └── dataimport.py
    ├── protocols
    │   ├── __init__.py
    │   ├── protocol_autopick.py
    │   ├── protocol_autopick_v2.py
    │   ├── protocol_base.py
    │   ├── protocol_center_averages.py
    │   ├── protocol_classify2d.py
    │   ├── protocol_classify3d.py
    │   ├── protocol_create_mask3d.py
    │   ├── ...
    ├── tests
    │   ├── __init__.py
    │   ├── test_convert_relion.py
    │   ├── test_protocols_relion.py
    │   └── test_workflow_relion.py
    ├── viewers.py
    └── wizards.py

Standard submodules

Scipion plugins are usually composed by the following submodules:

__init__.py

This file is required in Python modules and it is executed when the module is imported. Additionally, a Plugin class should be defined (subclass from pyworkflow.em.Plugin) and it should be registered in the Domain to define this module as a Scipion plugin.

TODO: explain in DETAIL what needs to go in init.py This has been copied from the last docs and needs to be reviewed and extended

define the Plugin class
Defining variables and paths?
Implement validateInstallation() (Optional)
Defining the plugin binaries (Optional)

The next step is to add the code responsible for the installation of the binaries. We redefine defineBinaries in our RelionPlugin class in the init.py. Please note how we have added default=True to the latest relion binaries. - this means that this binary will be installed automatically when we get this plugin (unless specified otherwise).

# this goes inside class RelionPlugin(Plugin):
    def defineBinaries(self, env):
relion_commands = [('./INSTALL.sh -j %d' % env.getProcessors(),
                          ['relion_build.log',
                           'bin/relion_refine'])]

env.addPackage('relion', version='1.4',
               tar='relion-1.4.tgz',
               commands=relion_commands)

env.addPackage('relion', version='1.4f',
               tar='relion-1.4_float.tgz',
               commands=relion_commands)

# Define FFTW3 path variables
relion_vars = [('FFTW_LIB', SW_LIB),
               ('FFTW_INCLUDE', SW_INC)]

relion2_commands = [('cmake -DGUI=OFF -DCMAKE_INSTALL_PREFIX=./ .', []),
                    ('make -j %d' % env.getProcessors(), ['bin/relion_refine'])]

env.addPackage('relion', version='2.0',
               tar='relion-2.0.4.tgz',
               commands=relion2_commands,
               updateCuda=True,
               vars=relion_vars)

env.addPackage('relion', version='2.1',
              tar='relion-2.1.tgz',
              commands=relion2_commands,
              updateCuda=True,
              vars=relion_vars,
              default=True)

bibtex.py

This submodule is not supposed to be imported directly, it should contain the bibtex string literal as the Python doc string. Scipion will take care of parse the bibtex reference and incorporate into the plugin module. TODO: ADD DETAILED EXAMPLE

constants.py

This submodule should contain all the constants that can be later imported in protocols etc. If there are only few of them, there is no need for a separate constants.py file. TODO: ADD DETAILED EXAMPLE

convert

This submodule might contain two files: convert.py with all functions used for conversion between base classes and programs inside the plugin; dataimport.py with import classes that are used in pyworkflow/em/protocol/protocol_import/. In cases when there are only few conversion functions, the submodule folder can be replaced by a single convert.py file. TODO: ADD DETAILED EXAMPLE

protocols

In this submodule all the protocols of the plugin should be implemented. Usually a plugin provides many protocols, so the most common case is to have a submodule folder with its own __init__.py and one .py file per each protocol class. TODO: ADD DETAILED EXAMPLE, MAYBE LINK IN ANOTHER PAGE?

viewers

A plugin can also define viewers for existing objects or new protocols. Since many built-in viewers are provided by Scipion, plugins do not define many viewers, so a viewers.py will serve as submodule. TODO: ADD DETAILED EXAMPLE, MAYBE LINK IN ANOTHER PAGE?

wizards

Wizards need to be defined for protocols/parameters, but many base classes are already provided. Here again the wizards.py submodule is usually enough. TODO: ADD DETAILED EXAMPLE, MAYBE LINK IN ANOTHER PAGE?

tests

We strongly recommend to follow Test-Driven-Development, so this is the place where all plugin tests should go. It is important to create different test cases from the beginning of the plugin development. TODO: ADD DETAILED EXAMPLE, MAYBE LINK IN ANOTHER PAGE?

logo.png

.gitignore

this file is required for Git. Here is an example:

#### Eclipse and so on
.project
.cproject
.pydevproject
.classpath
.idea

#### Python
build/
dist/
*.egg-info/
*.egg
*.py[cod]
__pycache__/
*.so
*~

These files are required for PyPI distribution. More information can be found here and examples of the contents of these files can be found in Publishing the Plugin to PyPI

  • LICENSE: license file for plugin code

  • CHANGES.txt: version history

  • README.rst: long description of your plugin (for PyPI catalog)

  • MANIFEST.in: includes links to README and LICENSE files

  • setup.py: this is a build script for PyPI distribution, containing important information about your plugin. See below for an example of such file.

Testing

  • In your terminal, add the plugin directory to PYTHONPATH. This will make our plugin available as a python module when we launch scipion. While we develop and change our code, every time we launch Scipion we will have our changes available.

    export PYTHONPATH=/path/to/scipion-em-relion
  • Check if all submodules are imported correctly

    scipion run python scripts/inspect_plugins.py relion
  • List your tests and copy the one you want to run:

    scipion test --show --grep relion

Publishing the Plugin to PyPI

We’ll explain below the steps followed to convert the package into a pip module that we can upload to pypi. Most of these steps are not scipion-specific, so it is recommended to check an external source (like this one or this one ) if you have doubts about pip packaging.

Adding PyPI files

We add four files to the folder scipion-em-relion: CHANGES.txt, setup.py, MANIFEST.in, README.rst.

setup.py

This is the most important one. It needs to call the setup function with, at least, the required arguments. Here we present a synthesized version:

  from setuptools import setup, find_packages
  from codecs import open
  from os import path
  
  here = path.abspath(path.dirname(__file__))
  
  # Get the long description from the README file
  with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
      long_description = f.read()
  
  setup(
      name='scipion-em-relion',  # Required
      version='1.0.0a',  # Required
      description='A python wrapper to use relion within Scipion',  # Required
      long_description=long_description,  # Optional
      url='https://github.com/scipion-em/scipion-em-relion',  # Optional, but very important
      author='Relion authors',  # Optional
      author_email='some@human.com',  # Optional
      keywords='scipion cryoem imageprocessing scipion-1.2',  # Optional
      packages=find_packages(),
      package_data={  #!!!!!! Required if we have a logo!!!!!
         'relion': ['logo.png'],
      }
  
  )

CHANGES.txt (optional)

This file records a short description of the modifications made with each release of the pip package.

v1.0.0, 23/04/2018 -- First commit

MANIFEST.in (optional)

The MANIFEST.in is needed so that our .txt file is included when we do the distribution (or many other non .py extensions, please check the official pip packaging guide if you need to include such files). IMPORTANT: if you have non-python files like images (except the logo), docs, scripts, you have to specify them here, otherwise they will be excluded from PyPi distribution! An example below recursively includes all files in spider/scripts folder. See more info here.

include *.txt
recursive-include spider/scripts *

Also, you will need to add/uncomment the following line into setup.py: include_package_data=True

Installing via pip locally

Remove the previous installation from Scipion

Remove binaries - if it applies. If you didn’t have a prior binary installation (i.e. you’re building this plugin new from scratch), go to next step.

rm -rf $SCIPION_HOME/software/em/relion*

Get plugins.json

Scipion requests a json list of available plugins from http://scipion.i2pc.es/getplugins and uses metadata from pypi to filter which packages are available for the current Scipion version. Since we want to test our pip plugin before we upload it to pypi, we will read locally a file like the one provided in the website, with our plugin added.

  • In a directory of your choice, add a plugins.json file with the appropriate info for your plugin - you can save Scipion’s plugins.json and add your plugin’s data.

    plugins.json:

    {
        "scipion-em-relion": {
            "name":"relion",
            "pipName": "scipion-em-relion",
            "pluginSourceUrl":"/path/to/your/scipion-em-relion"
        }
    }

    Note that when you add the key pluginSourceUrl, Scipion will use pip to install the plugin from that directory (i.e. pip will copy the relion folder to python’s site-packages folder). If this key is missing, Scipion will try to install from https://pypi.org/. Once you do this installation, changes made in your development folder will NOT be present in the copy used by Scipion. You would have to uninstall and go back to development mode using the variable PYTHONPATH.

  • In the VARIABLES section of your ~/.config/scipion/scipion.conf, add variable SCIPION_PLUGIN_JSON. If you don’t add this variable, Scipion will read the json from http://scipion.i2pc.es/getplugins instead of reading your local json copy. If you use pycharm to run Scipion, you can also add it as environment variable in your run configuration. Remember to replace the example provided with the right path:

    [VARIABLES]
    SCIPION_NOTES_PROGRAM =
    SCIPION_NOTES_ARGS =
    SCIPION_NOTES_FILE = notes.txt
    SCIPION_NOTIFY = False
    SCIPION_PLUGIN_JSON=/home/desktop/yaiza/plugins.json

Installation script

Scipion has a script to handle plugin installation / uninstallation. Use this script in a new terminal or reset the PYTHONPATH variable that we defined at the beginning. We have a few choices (un)installation choices:

  • Installing plugin and default binaries:

    $SCIPION_HOME/scipion installp -p scipion_grigoriefflab
    This command does two things:
    1. Gets the package from pypi

    2. Installs the default binaries (those that had default=True in the registerPluginBinaries function). If no errors happen, we get an output similar to this one:

      /home/yaiza/git/scipion/software/bin/python /home/yaiza/git/scipion/scipion installp -p scipion-em-relion
      
      Scipion  pluginization_install_config (2018-04-11) 0ee533a
      
      python  /home/yaiza/git/scipion/install/install-plugin.py /home/yaiza/git/scipion/scipion installp -p scipion-em-relion
      Building scipion-em-relion ...
      python /home/yaiza/git/scipion/software/lib/python2.7/site-packages/pip install /home/yaiza/git/scipion-em-relion
      Processing /home/yaiza/git/scipion-em-relion
      Installing collected packages: scipion-em-relion
        Running setup.py install for scipion-em-relion: started
          Running setup.py install for scipion-em-relion: finished with status 'done'
      Successfully installed scipion-em-relion-1.0a0
      Done (1.01 seconds)
      [. . .]
      Building relion-2.1 ...
      ...Relion binaries installation log goes here
      ...
      Done (0.20 seconds)
      
      Process finished with exit code 0
  • Uninstalling plugin and all binaries installed

    $SCIPION_HOME/scipion installp -p scipion-em-relion
  • We can use the flag --noBin to both install and uninstall without binaries:

    $SCIPION_HOME/scipion installp -p scipion-em-relion --noBin
  • Install specific plugin binaries (only works if we have done installp first).

    $SCIPION_HOME/scipion installb relion-2.1
  • Uninstall specific plugin binaries

    $SCIPION_HOME/scipion uninstallb relion-2.0

Testing

  • With your plugin and binaries installed, it is recommended to run some of your plugin’s tests to check everything is in order:

    scipion test em.packages.relion.tests.test_***
  • Open the test project:

    scipion last
    First, inspect the protocol output to make sure there’s nothing weird; then, open the protocol box to see if our logo and references are there. It’s important to do this step because if we don’t open the GUI we won’t be able to detect logo related issues.

Add your own DataSet

If you need an additional dataset you can do this and host it where ever you want /can. Let’s assume you need a new dataset…​

  • usually you work first locally until you are happy with your data set.

  • Decide where to host it and upload it. For that scipion will:

  • Generate a MANIFEST file

  • rsync it to your server, you will need to provide a login info (like user@server.com, and a remote folder location.

  • type something like:

    scipion testdata --upload myplugin123_testdata -l user@server.com -rf /path/at/the/server/for/your/datasets 

    Please note that the dataset name must be unique, so better prefix it with the plugin name. -l is the login for your server and -rf is the remote folder where to rsync your files.

  • Refer to if in your tests, at you tests folder/init.py:

    DataSet(name='myplugin123_testdata', 
            folder='myplugin123_testdata',
            files={
                   'file1': 'file1.txt'
                   ...},
            url='http://wwww.server.com/datasets')
    NOTE: url parameter should be a valid url where your dataset is being published. TIP: I haven’t tried, but doing the upload yourself, to generate the MANIFEST and then adding your datasets + MANIFEST to github might also work if you later point to the gitraw url?? (disclaimer…​I haven’t tried it.)

Create and upload distribution

To upload your distribution to pypi, you’ll need to create an account.

  • Install twine if you don’t have it

    pip install twine
  • Create the source distribution (at least! You can also create a Built distribution. Read more in the official [packaging guide]())

    cd $PLUGIN_HOME
    python setup.py sdist
    It is convenient to check your *egg-info/SOURCES.TXT and see if you miss any file (pay special attention to non-python files that you might have forgot to include in MANIFEST.in or in your setup.py, like the logo).
  • Upload the distribution WITH EARLIEST COMPATIBLE SCIPION VERSION IN THE COMMENTS.

    cd $PLUGIN_HOME && twine upload dist/* -c "scipion-1.2"
    This means that this release we’re uploading will be available for Scipion version 1.2 or higher. The scipion version must follow the pattern used above (scipion-X.Y(.Z)) Now our plugin is on [pypi](https://pypi.org/project/scipion-em-relion/).

Install from pip

  • Uninstall plugin:

    $SCIPION_HOME/scipion uninstallp -p scipion-em-relion
  • Remove SCIPION_PLUGIN_JSON from ~/.config/scipion/scipion.conf (we exit development mode)

  • Install

    $SCIPION_HOME/scipion installp -p scipion-em-relion
  • Test again

    scipion test em.packages.relion.tests.test_***
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.