added databases

cleaned up some markdown

celegans_herm/README.md

@@ -1,6 +1,8 @@
 Anterior Herm. C. Elegans Connectome
 ====================================
 
+The database is ```celegans_herm.db.gz```
+
 The connectome of the anterior section of the hermaphrodite c. elegans. The source
 of this data is from the [second listed dataset](http://www.wormatlas.org/neuronalwiring.html#NeuronalconnectivityII) at wormatlas. Quoting from them:
 
@@ -35,9 +37,8 @@ Synapses; note that the electrical synapses are undirected, the chemical synapse
 * count : # of this kind of synapse
 
 
-
 TODO
 ------
-Add "gross types"/ role from metadata
-Diagnostic plots
+- [ ] Add "gross types"/ role from metadata
+- [ ] more Diagnostic plots
 

celegans_herm/preprocess.py

@@ -6,6 +6,7 @@
 import os
 from ruffus import * 
 import dbmodel
+import subprocess
 
 DATA_DIR = "../../data/celegans/conn2"
 
@@ -226,6 +227,11 @@ def populate(infile, outfile):
                             Synapses.update(count = Synapses.count + nbr).where(Synapses.from_id == c1, 
                                                                                 Synapses.to_id == c2, 
                                                                                 synapse_type == st_short)
+    dbmodel.db.close()
 
+@files(populate, dbmodel.DB_NAME + ".gz")
+def compress_db(infile, outfile):
+    subprocess.call(["gzip", infile, outfile])
 
-pipeline_run([read_data, sanity_check, populate])
+pipeline_run([read_data, sanity_check, populate, 
+              compress_db])

mouseretina/README.md

@@ -1,6 +1,8 @@
 Mouse Retina Connectome
 ============================
 
+The database is ```mouseretina.db.gz```
+
 This is data from [Connectomic reconstruction of the inner plexiform
 layer in the mouse
 retina](http://www.nature.com/nature/journal/v500/n7461/full/nature12346.html)
@@ -28,39 +30,39 @@ Schema
 
 Table of cell ID and type
 
-* cell_id: globally-unique cell ID, from original paper ID
-* type_id : type_id, references Types table
+* ```cell_id```: globally-unique cell ID, from original paper ID
+* ```type_id```: type_id, references Types table
 
 ### SomaPositions
 
 The locations of the soma of the cells, from original plots, 
 via image reconstruction. Only available for 950 cells. 
 
-* cell_id: relates to cells 
-* x : position (um)
-* y : position (um)
-* z : position (um )
+* ```cell_id```: relates to cells 
+* ```x``` : position (um)
+* ```y``` : position (um)
+* ```z``` : position (um )
 
 ###Contacts
 
 The locations of the cell-cell contact points (this is validated between xls and the matlab file). Remember the edges are undirected, so from/to are arbitrary
 
-* from_id: cell id of first cell
-* to_id: cell id of second cell
-* x : position of contact (um)
-* y : position of contact (um)
-* z : position of contact (um)
-* area : area of contact (um^2)
+* ```from_id```: cell id of first cell
+* ```to_id```: cell id of second cell
+* ```x``` : position of contact (um)
+* ```y``` : position of contact (um)
+* ```z``` : position of contact (um)
+* ```area``` : area of contact (um^2)
 
 
 ### Types
 
  Cell Type metadata, reverse-engineered from their plot; only covers
 types 1-71 (why are there more types? what are they?)
 
-* type_id : type ID
-* designation: the authors' original string designating the cells
-* Volgyi_volgi : type from CITE
-* MacNeil: Type from Macneil
-* certainty : The author's arbitrary uncertainty metric of L, M, H when relating the cell to one of the known types
-* coarse : the coarse type, determined from supplemental text
+* ```type_id``` : type ID
+* ```designation```: the authors' original string designating the cells
+* ```Volgyi``` : type from CITE
+* ```MacNeil```: Type from Macneil
+* ```certainty``` : The author's arbitrary uncertainty metric of L, M, H when relating the cell to one of the known types
+* ```coarse``` : the coarse type, determined from supplemental text

mouseretina/preprocess.py

@@ -16,6 +16,7 @@
 import skimage.measure
 import skimage.io
 import dbmodel 
+import subprocess
 
 LIGHT_AXIS = [0.9916,0.0572, 0.1164]
 MAX_DIM = [132.0, 114.0, 80.0]
@@ -275,7 +276,8 @@ def create_db(_, outfile):
 @follows(type_metadata)
 @follows(merge_positions)
 @follows(create_db)
-def populate_db():
+@files(create_db, "dbdone.setinel")
+def populate_db(infile, outfile):
     dbmodel.db.connect()
 
     from dbmodel import Types, Cells, SomaPositions, Contacts
@@ -319,7 +321,9 @@ def populate_db():
                     Contacts.create(from_id = from_id, to_id = to_id, 
                                     x = r['x'], y=r['y'], z=r['z'], 
                                     area = r['area']).save()
-
+    dbmodel.db.close()
+    open(outfile, 'w').write(" ")
+
 @follows(transform_data)                                                      
 @files(["conn.areacount.pickle", "xlsxdata.pickle"], 
        "adj_comp.pdf")
@@ -361,6 +365,10 @@ def mat_xls_consistency((conn_areacount, xlsdata), adj_plots):
     f.savefig(adj_plots)
 
 
+@follows(populate_db)
+@files(dbmodel.DB_NAME, dbmodel.DB_NAME + ".gz")
+def compress_db(infile, outfile):
+    subprocess.call(["gzip", infile, outfile])
 
 
 if __name__ == "__main__":
@@ -370,6 +378,7 @@ def mat_xls_consistency((conn_areacount, xlsdata), adj_plots):
                   mat_xls_consistency, 
                   process_image_pos, merge_positions, type_metadata, 
                   create_db, 
-                  populate_db
+                  populate_db, 
+                  compress_db
               ])