The API (application program interface) for ipyrad is a way of directly accessing the nuts and bolts of ipyrad using Python. This has a number of advantages over the CLI (command line interface) in that there is greater flexibility for creating highly complex branching assemblies, or for applying ipyrad in a non-standard way. Perhaps the best feature, though, is that you can perform entire analyses within Jupyter notebooks<notebooks> to create documented reproducible code for your analyses.
If you are using the API then you must have an ipcluster instance started in order to parallelize your code. This can be started locally by opening a separate terminal and running (ipcluster start -n=10) to start 10 engines. Or, to run your code on a remote cluster set up your ipcluster instance following this tutorial.
- Assembly -- perform all assembly steps available in the CLI but with several additional options.
* Analysis -- analyze and compare the size and distribution of data sets, create plots, calculate population genetic statistics, and perform phylogenetic analyses.
Our goal in developing the ipyrad API is not only to get people writing Python scripts, but also to encourage the use of an exciting new tool called Jupyter notebooks, an excellent tool for reproducible science.
The envisioned usage of the ipyrad Python API is to run test assemblies within a Jupyter notebook on a local computer. Once you've tested that your assembly looks good, and that your selected parameters seems appropriate you can then take your script and submit a long running job to a larger computing cluster.
Beyond the very easy way that the CLI provides for connecting to HPC clusters, the API provides even more flexibility for doing very advanced computation for users that are fluent with the ipyparallel package.
Assembly and Sample objects are used by ipyrad to access data stored on disk and to manipulate it. Each biological sample in a data set is represented in a Sample object, and a set of Samples is stored inside an Assembly object. The Assembly object has functions to assemble the data, and stores a log of all steps performed and the resulting statistics of those steps. Assembly objects can be copied or merged to allow branching events where different parameters can subsequently be applied to different Assemblies going forward. Examples of this are shown below.
To create an Assembly object call ip.Assembly() and pass a name for the data set. An Assembly object does not initially contain Samples, they will be created either by linking fastq files to the Assembly object if data are already demultiplexed, or by running step1() to demultiplex raw data files, as shown below.
ipyrad_scripts.rst pedicularis.rst HPC_script.rst
Begin by creating an Assembly class object, this is equivalent to running the (-n) argument from the command line. It initializes an the Assembly object that will be used to store the assembly parameters that will be used.
## initalize an Assembly class object
data = ip.Assemble("test")The assembly object has a number of attributes and functions that you can access interactively in IPython by using tab-completion. Simply type the name of your object (here named 'data'), followed by tab to see all of the options.
data.<tab>
data.barcodes data.dirs data.name
data.branch data.files data.outfiles
data.build_stat data.get_params data.paramsdict
data.clust_database data.link_fastqs data.populations
data.database data.link_populations data.run Two important functions of the Assembly include get_params() and set_params(), which are used to view and modify the parameter settings, respectively.
data.get_params()0 assembly_name test 1 project_dir ./ 2 raw_fastq_path 3 barcodes_path 4 sorted_fastq_path 5 assembly_method denovo 6 reference_sequence 7 datatype rad 8 restriction_overhang ('TGCAG', '') 9 max_low_qual_bases 5 10 phred_Qscore_offset 33 11 mindepth_statistical 6 12 mindepth_majrule 6 13 maxdepth 10000 14 clust_threshold 0.85 15 max_barcode_mismatch 0 16 filter_adapters 0 17 filter_min_trim_len 35 18 max_alleles_consens 2 19 max_Ns_consens (5, 5) 20 max_Hs_consens (8, 8) 21 min_samples_locus 4 22 max_SNPs_locus (20, 20) 23 max_Indels_locus (8, 8) 24 max_shared_Hs_locus 0.5 25 edit_cutsites (0, 0) 26 trim_overhang (4, 4, 4, 4) 27 output_formats * 28 pop_assign_file
You can change parameters by providing either the name or index number of the parameter you want to change and the new parameter setting to set_params. Below we also show what happens if you enter an invalid parameter.
## change a few parameters
data.set_params("project_dir", "iptest")
data.set_params("raw_fastq_path", "ipsimdata/rad_example_R1_.fastq.gz")
data.set_params("barcodes_path", "ipsimdata/rad_example_barcodes.txt")
## if you enter a parameter setting that is invalid an error will raise
data.set_params("max_alleles_consens", "four")- IPyradError:
Error setting parameter 'max_alleles_consens' invalid literal for int() with base 10: 'four' You entered: four
The next important function is the run() command, which is used to run steps of the assembly.
## run a few steps of the assembly
data.run("12")Let's imagine at this point that we are interested in clustering our data at two different clustering thresholds. We will try 0.90 and 0.85. First we need to make a copy the Assembly object. This will inherit the locations of the data linked in the first object, but diverge in any future applications to the object. Thus, they can share the same working directory, and will inherit shared files, but create divergently linked files within this directory. You can view the directories linked to an Assembly object with the .dirs argument, shown below.
## run step 3 to cluster reads within samples
data.run(3)
## print the results
print data1.stats
## create a new branch
data2 = data.branch("c85")
data2.set_params("clust_threshold", 0.85)
## run steps on new assembly
data2.run(3, force=True)And you can see below that the two Assembly objects are now working with several shared directories (working, fastq, edits) but with different clust directories (clust_0.85 and clust_0.9).
print "data1 directories:"
for (i,j) in data.dirs.items():
print "{}\t{}".format(i, j)
print "\ndata2 directories:"
for (i,j) in data2.dirs.items():
print "{}\t{}".format(i, j)