The Rewrite: BB2 Redux

[zeffii]

Over the next couple of days (unless something unforeseen happens) I have time to make a start with building the add-on from scratch. Getting it to match the working features of BB2 will take a while. It's difficult to put a time frame on it until the process gets rolling.

The decision to start from scratch is not done in haste. I think some of the responders on this repository have expressed similar feelings about how to proceed. There's lots that can be taken from the current code-base, but a lot that I really want to sanitize and arrange into separate files. The core of this was written when Blender 2.5 was still young and the API not very well defined, and prone to convoluted solutions.

If I can reach feature parity and structure the code better then I will consider it a success, speeding up the import (and avoiding recursive updates) is not my first priority, making it easier for others to contribute is far more important.

Ticks indicates functionality seems to be restored for that panel, certainly report any issues with those sectons. The unticked sections have had no major checks done other than suppressing import errors. The section labelled next is either being worked on or that's where the most recent code changes have taken place.

• gui_panel_ep.py
• gui_panel_mlp.py
• gui_panel_nma_viz.py
• gui_panel_output_animation.py
• gui_panel_pdb_import.py
• gui_panel_pdb_output.py <---next
• gui_panel_physics_sim.py
• gui_panel_view.py

[zeffii]

The plan. (might change as it progresses, and is not chronologic)

• All new code must be pep8 compliant, with the exception of the following error types --ignore=E402,E702,E501,E731,E241 (Use a pep8 linter!, else thyne code shalt not pass!). If you use tabs instead of 4 spaces for indentation you will be slapped with a fish.
  • pep-20 / Zen of Python helps you decide how to do things.
  • http://docs.quantifiedcode.com/python-anti-patterns/index.html
• Generate the default scene and objects using a script, unless it really can't be done any other way
• Generate new documenation folder with layout something like

BioBlender/Manual/images/01_01_Intro.png
BioBlender/Manual/images/01_02_Intro_p2.png
BioBlender/Manual/images/02_01_Install_Guide_step1.png
BioBlender/Manual/images/02_01_Install_Guide_step2.png
BioBlender/Manual/images/02_01_Install_Guide_step3.png
BioBlender/Manual/Introduction.md
BioBlender/Manual/Installation.md
BioBlender/Manual/Feature_01.md  <-- replace number with shortname for feature
BioBlender/Manual/Manual.md  <-- would link all .md files together.

• Separate operators into own files
• Separate panels (ui) into own files
• Separate Tables
• Separate Utility code
• Employ OS agnostic python tools to deal with the double quote mayhem ( there's a dedicated core python module for it shlex
• Create disposable non rigged import for fast viewing (already coded..not merged)
• Parent the EP Curves to mainting outliner sanity. Make sure it works with animated EP

[MonZop]

Great plan!
I am certainly willing to take care of the documentation folder (help always welcome, of course, but if you are a coder it might be more useful in other places), which is the only part that I understand, almost. uh

[zeffii]

I like a fresh start, unfortunately there won't be anything to test for a while.. it's like a fetus now

[zeffii]

I made a new branch and split the panels into their own source files. This is no-where near working, but it's necessary until gradually things start to work again. It is my hope that I will have useful progress before the replacement dependency graph (aprox two weeks from now according to the developers meeting).

It has been difficult to find motivation to do this lately, but i'd like to think I can finish what I started.

[zeffii]

panels are now separated, and load in the correct sequence. <still non functional>

now to start linking up the global storage dictionaries again. This is the part I'm not terribly sure what kind of time it will take. But already in the last 2 hours this has gone faster than I anticipated. Back to coding.

[MonZop]

Good to hear things are going!
Just let me know whan (and if) I can do something, like testing

[zeffii]

it would help me a lot if you found time to learn how to use GIT from the command-line, or find someone to teach you. Even if you aren't coding anything it helps if i'm confident that the files you are testing are indeed the branch i'm working on.

There are only a few commands that anyone needs to know

git pull --all
git push --all
git add --all
git commit -a -m "commit message here"
git checkout -b <branch_name>  # new_branch_name_based_on_current_branch
git branch -D <branch_name>
git branch

[MonZop]

OK, I'll make an effort, (find someone to help me tomorrow, if available).
I already know that once learned, it will be easy, but i've been lazy, and had also much to do.
This Sunday (10 of May) I'm going to Munich to the Premiere of the film The Dark Gene, which contains several minutes of an animation made by us using BioBlender! Munich DOK.fest
I'm looking forward to see how the poublic will react at those images!
(I mean, besides Blender people, and biologists)

[zeffii]

so far so good, but won't know how easy the rest will be until the next push.

Cool, a premiere! Must be great to finally see your work in context, enjoy!

[zeffii]

This is much easier to navigate now. wow.

I think the import panel has reached feature parity with older versions, but it has not been tested extensively yet. The import panel is 99% existing code, and small changes to make things Blender version agnostic. I've replaced boolean and integer global variables with scene specific properties which are by definition global. Added aliasing to avoid 'dot lookups' where possible.

Time to continue to the next panel. MLP. because I know it a little bit.

[zeffii]

mlp surface works now:

still too rough for testing..

[zeffii]

render mlp to surface now too:

[MonZop]

Beautiful!
Congratulations!
I have also tried (using the redux_s3_FIRE) , and got the MLP on the surface, but not the render.
I also have a question: is this MLP the value calculated by MLP.py, or is it the color of pyMOL, simply transformed to grey scale, and rendered?

[zeffii]

I have also tried (using the redux_s3_FIRE) , and got the MLP on the surface, but not the render.

i'll look at it. The code feels a little like this now : screenshot from 2015-05-08 14 46 23

there's a conversion function that strips out the colour, will look carefully at what it does.

[MonZop]

WOW, it makes me feel dizzy with vertigo!

[zeffii]

At present:

I haven't yet traced manually what they do, technically.

[zeffii]

(wrong button pressed closed the thread)

here's the abstract of those two functions, ps: this might take a while to fully write and will edit this post over the course of hours/days. I have made no functional changes to any of this code, except minor formatting (may 2015).

bb2_operator_mlp()

In theory:

• expects user to select an atom of a model, any atom.
• if multiple atoms are selected, its looks to see if the atoms are from one .pdb or multiple
• for every pdb model identified by the selection it creates a tID variable
  • perform the mlp(tID) function
  • todoAndViewpoints()

bb2_operator_mlp_render()

• requires the surface, (or surfaces if multiple) to be selected..
• it finds the selected PDB ids and gives them an ID, tID (the ID is used for internal reference, it doesn't reflect the pdb file name..)
• for every found tID,
  • call mlpRender(tID)
  • todoAndViewpoints()

mlp()

..todo

mlpRender()

..todo

[zeffii]

While reading the operator code, I'm wondering if it wouldn't be tidier if there was a UI list, which shows the currently imported models, and lets the user select them from there, rather than the atoms. I'll read up on how to do a UI list, it's something i've wanted to know for a while anyway.

[MonZop]

Several issues here:

• A list of objects (elements from one or more pdb files) would be great. I thought that this could be done (more) easily in the Outliner, so that every object would be associated with its atoms, its surface, its UV map (from MPL calculation) and its set of lines (from EP).
• BB2_operator_mlp() is supposed to:
  • get the molecular surface
  • run pyMLP.py using data from the pdb file, and fi values stored
  • use the .dx grid generated by pyMLP.py to attribute the value to every vertex of the surface
  • convert those values to grey scale
  • build a .obj file that lists the vertices as x,y,z and mlp
• BB2_operator_mlp_render(), will use the surface, that now has a UV map, and apply the render procedure that makes it rough/dark/dull or smooth/white/shiny according to the mlp value.

Hope this hleps!

[zeffii]

on reflection yes, using the outliner alone is fine for atom selection

i'm not sure about this question:

I also have a question: is this MLP the value calculated by MLP.py, or is it the color of pyMOL, simply transformed to grey scale, and rendered?

My code changes shouldn't have changed any operational features, are you asking me about the internals or has something changed due to my edits?

[zeffii]

mlp()

Quoted text by monzop as response to my (zeffii) code observations.

1. first determines the force formula (dubost,...buckingham)

   Yes, default formula is Testa

2. if posix based (linux, osx), then chmods pyMLP.py (gives it permission to be run)

3. then launches this nightmare of a string concatenation (this is one line) . Which I think means "get from tmp.pdb a tmp.dx"

   command = "%s %s -i %s -m %s -s %f -o %s -v" % (quotedPath(pyPath), quotedPath(homePath + "bin" + os.sep + "pyMLP-1.0" + os.sep + "pyMLP.py"), quotedPath(homePath + "tmp" + os.sep + "tmp.pdb"), method, spacing, quotedPath(homePath + "tmp" + os.sep + "tmp.dx"))

   I cannot read (language unknown to me), but the tmp.dx is a file which is generated by pyMLP. One pssiblity is that pyMLP uses a file already prepared, and fills in witht he values. Pls. note that pyMLP needs all the fi values, which are stored as values_fi, by aminoacid and by atom, in the very big BioBlender2.0.py code,(Lines 186 to 224).

4. then it calls surface()

   1. opens tmp.pdb to correct 1+ 1- errors (whatever those are)
   2. generates tmpPath0 a surface.pml which seems to be a script that pyMOL can read
   3. launched pyMOL with
      python command = "%s -c -u %s" % (quotedPath(pyMolPath), quotedPath(homePath + "tmp" + os.sep + "surface.pml"))
   4. imports a tmp.wrl file as an object and names it "SURFACE"
   5. removes doubles, shades smooth, translates and rotates the surface to match the rotation of the Empty.

   if the wrl file also contains color infomration, this should be removed at this step
   if the user already generated a surface, it is not necessary to repeat the process here.

5. waits for the asynchronous process (3) to complete

6. then the .dx is opened and read and updates a number of variables dxData, dimension, origin, delta

7. if there was dxData , vertex_paint_mode is entered and again it removes doubles and shades smooth (this might be redundant)

   indeed, it seems redundant. it also seems odd that the removal of doubles and smoothing should be associated with vertex_paint_mode

8. then it reads the vertex colour map (which was generated during the wrl import (4. IV), for each vertex of each polygon in the color map it performs a modification to the vertex colours as imported.

   tmp = (color_map.data[i].color[0] + color_map.data[i].color[1] + color_map.data[i].color[2]) / 3
   color_map.data[i].color = [tmp, tmp, tmp]

   definitely forced greyscale

   • this is where we say (in italian) that the donkey falls. In english: error revealed.
     In step 4.IV all colors must be removed.
     The new colors are attributed to the verteces by trilinear interpolation of the values in the .dx file, to the corresponding vertex.
     this step never worked after BB0.6 because (the coder said) vertex identities were not accessible. *

9. then does a vertex_color_smooth on the whole map

10. sets viewport to TEXTURED

11. lastly it hides the atoms in the scene

These last steps do work. I am almost sure that if you do remove the colors from the sutface generated by pyMOL, the part of running pyMLP and attributing colors to verteces has never worked.
The way it used to work in early versions was by calling a script called OBJcreator, which was capable of putting together the info from the surface (.wrl) and the MLP (.dx) and created a new file with colors (.obj)

[zeffii]

mlpRender()

todo

[zeffii]

@MonZop i've formatted that last post so it is more clear to me (and readers) who is commenting what.

[zeffii]

OK, i'll check out OBJCreator script

[zeffii]

the .dx file is a massive list of 3 Floats per line (I doubt these are colours?)
the .wtl is imported using a import_x3d provided by blender, it contains verts+faces+colours per vertex

In the end where do you want the colour to come from?

[zeffii]

wait... the dx file is a grid ? a representation of values in a 3d vector space? So is the idea to make the polygons on the 'SURFACE' object mesh to the nearest value in the dx file?

[zeffii]

like this?

[zeffii]

object 1 class gridpositions counts 43 40 34
origin -2.213350e+01 -1.778850e+01 -1.658250e+01
delta 1.000000e+00 0.000000e+00 0.000000e+00
delta 0.000000e+00 1.000000e+00 0.000000e+00
delta 0.000000e+00 0.000000e+00 1.000000e+00
object 2 class gridconnections counts 43 40 34
object 3 class array type double rank 0 items 58480 data follows

suddenly makes more sense: 43 * 40 * 34 == 58480

[zeffii]

'''
Define atom scales
0 vdw = visual Van der Waals scale
1 cov = visual covalent scale
2 collision = collision radius scale
3 color = representation 
'''
atom_properties = {
    # :   vdw, cov, collision, color
    'C':  [1.35, 1.10, 0.6, [0.10, 0.10, 0.10]], 
    'CA': [1.59, 0.99, 0.6, [0.40, 1.00, 0.14]], 
    'N':  [1.23, 1.07, 0.6, [0.24, 0.41, 0.70]], 
    'O':  [1.20, 1.04, 0.6, [0.46, 0.10, 0.10]], 
    'S':  [1.43, 1.46, 0.6, [1.00, 0.75, 0.17]],
    'P':  [1.43, 1.51, 0.6, [1.00, 0.37, 0.05]], 
    'FE': [1.59, 0.64, 0.6, [1.00, 0.50, 0.00]], 
    'MG': [1.37, 0.65, 0.6, [0.64, 1.00, 0.05]], 
    'ZN': [1.10, 0.74, 0.6, [0.32, 0.42, 1.00]], 
    'CU': [1.10, 0.72, 0.6, [1.00, 0.67, 0.00]],
    'NA': [1.80, 0.95, 0.6, [0.80, 0.48, 1.00]], 
    'K':  [2.18, 1.33, 0.6, [0.72, 0.29, 1.00]], 
    'CL': [1.37, 1.81, 0.6, [0.10, 1.00, 0.60]], 
    'MN': [1.59, 0.46, 0.6, [0.67, 0.60, 1.00]], 
    'H':  [0.95, 0.53, 0.3, [0.90, 0.90, 0.90]], 
    'F':  [1.16, 1.36, 0.6, [0.27, 0.80, 0.21]]
}

[zeffii]

the above table is not updated from data in your table (yet!), merely the three old tables merged.

[MonZop]

atom_properties = {
    # :   vdw, cov, collision, color
    'C':  [1.70, 0.70, 0.6, [0.10, 0.10, 0.10]], 
    'CA': [2.30, 1.80, 0.6, [0.40, 1.00, 0.14]], 
    'N':  [1.55, 0.65, 0.6, [0.24, 0.41, 0.70]], 
    'O':  [1.52, 0.60, 0.6, [0.46, 0.10, 0.10]], 
    'S':  [1.80, 1.00, 0.6, [1.00, 0.75, 0.17]],
    'P':  [1.80, 1.00, 0.6, [1.00, 0.37, 0.05]], 
    'FE': [1.30, 1.30, 0.6, [1.00, 0.50, 0.00]], 
    'MG': [1.73, 1.30, 0.6, [0.64, 1.00, 0.05]], 
    'ZN': [1.40, 1.30, 0.6, [0.32, 0.42, 1.00]], 
    'CU': [1.40, 1.40, 0.6, [1.00, 0.67, 0.00]],
    'NA': [2.27, 1.60, 0.6, [0.80, 0.48, 1.00]], 
    'K':  [2.75, 2.00, 0.6, [0.72, 0.29, 1.00]], 
    'CL': [1.75, 1.00, 0.6, [0.10, 1.00, 0.60]], 
    'MN': [1.40, 1.40, 0.6, [0.67, 0.60, 1.00]], 
    'H':  [1.20, 0.30, 0.3, [0.90, 0.90, 0.90]], 
    'F':  [1.47, 0.60, 0.6, [0.27, 0.80, 0.21]]
}

This table includes the more realistic sizes for VdW and Cov. I will work more on colors

[zeffii]

Cool. now the table exists, could implement it a.s.a.p. and make a new branch for the modified table, it doesn't have a massive impact on the code. maybe I'll delete / "stash"(archive) stages s1..s5, or delete them altogether.

[zeffii]

i've pushed, but not yet tested. I like the leading lines, normally we don't format 'dicts' in any special way, but because this is a table with a legenda, I think this convention is OK.

atom_properties = {
    #      vdw
    #       |    cov
    #       |     |  collision
    #       |     |     |          color
    #       |     |     |           |
    'C':  [1.70, 0.70, 0.6, [0.10, 0.10, 0.10]],
    'CA': [2.30, 1.80, 0.6, [0.40, 1.00, 0.14]],
    'N':  [1.55, 0.65, 0.6, [0.24, 0.41, 0.70]],
    'O':  [1.52, 0.60, 0.6, [0.46, 0.10, 0.10]],
    'S':  [1.80, 1.00, 0.6, [1.00, 0.75, 0.17]],
    'P':  [1.80, 1.00, 0.6, [1.00, 0.37, 0.05]],
    'FE': [1.30, 1.30, 0.6, [1.00, 0.50, 0.00]],
    'MG': [1.73, 1.30, 0.6, [0.64, 1.00, 0.05]],
    'ZN': [1.40, 1.30, 0.6, [0.32, 0.42, 1.00]],
    'CU': [1.40, 1.40, 0.6, [1.00, 0.67, 0.00]],
    'NA': [2.27, 1.60, 0.6, [0.80, 0.48, 1.00]],
    'K':  [2.75, 2.00, 0.6, [0.72, 0.29, 1.00]],
    'CL': [1.75, 1.00, 0.6, [0.10, 1.00, 0.60]],
    'MN': [1.40, 1.40, 0.6, [0.67, 0.60, 1.00]],
    'H':  [1.20, 0.30, 0.3, [0.90, 0.90, 0.90]],
    'F':  [1.47, 0.60, 0.6, [0.27, 0.80, 0.21]]
}

[zeffii]

now vdw looks like :

but covalent is unusual :) but i don't know if good/bad

[zeffii]

I guess, as a layman, the lower image makes more sense if I imagine tubes connecting the atoms to represent bonds..

[MonZop]

Oh oh, something wrong.
Let me double check the values

[MonZop]

But the VdW look right

[MonZop]

The covalent bond is (by definition) the contriobution of each atom to the total distance between two atoms: in theoryall atoms should touch their neighbours to which they are bonded.

I have two possible explanations:
a) there is a mistake and the covalent and collsion radii got mixed up
b) atom size is not set to precise size, but is scaled by a factor (less likely)

[zeffii]

weird :) i don't see anything immediately wrong with how I restructured the code, but it does seems weird that the game radius is larger than the covalent radius 0_o

[zeffii]

OK, i don't know :)

[zeffii]

The code should behave the same as before, but the big difference are the new values of the Covalent. I could for a test replace the cov scales values with the original ones, to see if I made a mistake in the scale assignments (it's not much effort to find out)

[zeffii]

btw, 'covalent scale' , what measurement is that relative to ? the radius of a H atom?

[MonZop]

I notice that in the Transform panel, (besides the fact that all parameters are keyframed, including Scale, which should not be), there is the Dimension value, and a Scale value..
I guess that the sphere are created with a radius, (possibily the Cov), and then scaled by the same value.
Example: Oxygen has a Scale = [0.6, 0.6, 0.6].
If I change the scaling to = [1.0, 1.0, 1.0] the resulting size is .. [1.3, 1.3, 1.3], which is not the right size anyways!!
wiered! 😕

[zeffii]

hmm, it may be that the cov values in the old table take this into account? It may just require a simple inversion..

[zeffii]

hehe, I raised this concern even before tackling the code: #32 (comment)

[MonZop]

Good point!
What if we: create the atom/sphere all with size 1 (radius), and then change to the proper radius, according to what we want .
It seems to me that even the VdW (which at first sight seems OK) has problems

[zeffii]

this is what gets imported as the 'template atom' at the moment:

[MonZop]

I would say that it is the typical Carbon atom, radius 0.7

[zeffii]

right, everything is scaled relative to the carbon atom then

[zeffii]

maybe figure this out today, I have some quiet time to myself.

[MonZop]

Oooops!! Checking the new Bledner, I was playing with MLP, and noticed that it does map perfectly on the surface, but the render has lost the texture and material features.
I cannot say if these features worked before, as the master cannot render the MLP.
Any hint?

[zeffii]

it does map perfectly on the surface, but the render has lost the texture and material features.

Which branch? I don't recall modifying a lot of code for the MLP panel, it is entirely possible that I didn't catch one or two things. I have the feeling that render / baking still relies on the values in one of the imported temporary files..and may just need to be told to read the new vertex colour data for baking noise.

Also i'm still not sure what exactly to do about the vdw / covalent scale flip, but the code to do it is in place

[zeffii]

I responded , but can't look at the code right now

[zeffii]

OK, I can return to this now, this bugs the hell out of me.

I'll write a small test script that generates an Atom of each type and provide a switch to flip between covalent vs vdw radius. This way the table and switching radii can be understood separately from the BioBlender addon. It doesn't make the slightest bit of difference if we are looking at Molecules or just a string of unique atoms in x,y space to get the right scaling.

I'd really like to move on from this bit of the code.

[zeffii]

or an artificial soup of molecules that show each type of atom, for quick debugging

[MonZop]

I have prepared a pdb file, called BunchOfAtoms, but 1. I do not remember how to add it to the Test Molecules folder and 2. I am a little ashamed at circulating a totally meaningless/impossible atomic file.
I can send it through the mail for your use

[zeffii]

yes, please send via email. it's a debug pdb then..so not entirely meaningless :)