Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducibility of Charmm results in original paper #4

Closed
ohsOllila opened this issue Mar 25, 2015 · 42 comments
Closed

Reproducibility of Charmm results in original paper #4

ohsOllila opened this issue Mar 25, 2015 · 42 comments

Comments

@ohsOllila
Copy link
Member

The order parameters for POPC acyl chain with CHARMM36 model are larger than in experiments in figure https://github.com/NMRLipids/NmrLipidsCholXray/blob/master/FIGS/OrderParametersCHOL.jpg.
The results look similar as reported by Piggot et al. [Fig. 5 in dx.doi.org/10.1021/ct3003157 | J. Chem. Theory Comput. 2012, 8, 4593−4609].
However, in the original CHARMM36 paper a better agreement with experiments is reported [Fig. 9 inhttp://dx.doi.org/10.1021/jp101759q, J. Phys. Chem. B, Vol. 114, No. 23, 2010]. This is similar difference as in area per molecule.

I think that it is necessary to find out what is the reason for this. Does CHARMM36 really have too condensed bilayer, or if there is something wrong in the Gromacs conversion?

@ClaireLoison
Copy link

Hi, I have launched several NAMD simulations of DPPC
charmm36 with various simulations parameters, to check their
impact on the order parameters of the heads and tails.

Here are some results on pure DPPC, in a simulation similar to the one published in
JCTC paper by Lee et al J. Chem. Theory Comput., 2016, 12 (1), pp 405–413. DOI: 10.1021/acs.jctc.5b00935, with the LJ 10-12 A cutoff.

I'll give some more results about other simulations settings soon.
BR, Claire


Pure DPPC (128 lipids and 8000 water) at 323K , 1 atm isotropic NPT , fixed Lx/Ly,
using NAMD (10-12 cutoff and vdwForceSwitching ON, and PMEInterpOrder 6)
trajectory of 170 ns, analysis of 150 ns.
The area per lipid of the present results is 62.05 +/- 0.17 A^2.
In Lee et al, JCTC2016, they publish a value of 61.6 +/- 0.1 A^2.


beta1 -3 0.06259640550975085 0.1777529934747562
beta2 -3 0.06684599168671689 0.17670897977293062
alpha1 -2 -0.037893429268013555 0.2104895547113332
alpha2 -2 -0.03392588629962274 0.20807051759744993
g31 -1 0.23304563799012307 0.09205662383799668
g32 -1 0.25076626561533394 0.08206788122407817
g2 0 0.19711614283905565 0.10792360336393667
g11 1 0.1794330398761552 0.12258764797662364
g12 1 0.04280008222918471 0.14626471474521555
sn2sn1 2 0.23270381622981884 0.04447526231786287 0.10476532070112063 0.06300468168885205
sn2sn1 3 0.18195641389178108 0.059036453288717906 0.21384231145614102 0.052740446227575635
sn2sn1 4 0.2101293128779962 0.05412861574685549 0.21809439655894627 0.05158248548481007
sn2sn1 5 0.20808129696325764 0.05473463945047225 0.2289410995571977 0.04989676658508056
sn2sn1 6 0.21947500134109799 0.052402573733679444 0.2234407694843929 0.05080569576392157
sn2sn1 7 0.20883366942545906 0.05468994921263106 0.22443842532627004 0.05141148579095683
sn2sn1 8 0.20728356674937778 0.05507073364665832 0.21661470565670274 0.053237387049344186
sn2sn1 9 0.19315350529910397 0.058024357552417784 0.21240360805243835 0.054668019094617284
sn2sn1 10 0.18938036431373168 0.05871116477087772 0.197890542230332 0.05772014839171035
sn2sn1 11 0.1728189005105111 0.06151741616002106 0.18880102591627212 0.05950104116369811
sn2sn1 12 0.16088731632765765 0.06339992699194952 0.17415063707490777 0.06149872484115409
sn2sn1 13 0.14171471476471886 0.06542989550547473 0.16192730234118824 0.0631775278591208
sn2sn1 14 0.1253987726773875 0.06651889044216003 0.13791559062128803 0.06554833458694857

sn2sn1 15 0.09253427959845012 0.06864963229660581 0.11180686965525233 0.06694917560353464

@ohsOllila
Copy link
Member Author

ohsOllila commented Sep 5, 2016

Thanks! I did calculate the acyl chain order parameters from this data: http://dx.doi.org/10.5281/zenodo.15549

They are, indeed, larger:
gromacsvsnamdsn1

gromacsvsnamdsn2

For area per lipid I get 60.2 Å^2

One detail attracted my attention; you have used 10-12 cutoff and vdwForceSwitching ON. In CHARMM36 paper (Klauda et al. 2010) it is written that:
"A LJ switching function over 8 to 12 Å was used in MD simulations with CHARMM. For the NAMD simulations, a shorter LJ switching function was used (11 to 12 Å)."
Are the reasons and consequences of these differences well known (maybe these are discussed in Lee et al., I have not yet red it carefully)?
In Gromacs we have (at least in the data used here):
rvdw = 1.2
rvdw_switch = 0.8

@jmelcr
Copy link
Contributor

jmelcr commented Sep 6, 2016

It had struck me that this effect can be easily produced by a barostat if misconfigured or if it has some glitches or implementation inaccuracies... The Areas per lipid are different and they might be the source of the difference in the OPs.

Right now, I'm running pure POPC (Charmm36) with openMM7.1 and Gromacs5.1.2 to see, whether there are differences -- and I'll probably try exchanging the barostat in Gromacs to see whether there's an effect...

@ClaireLoison
Copy link

Indeed, the barostat may play a role.

Also, sometimes in gromacs I think you can activate the

calculation of the correction to pressure due to the LJ cutoffs.

This has no sense in membrane since the assumption of a homogeneous

medium does not hold, but maybe there is a spurious default somewhere ?

If such correction would be inserted in the barostat, this may change the area/lipid, maybe.

Claire


Claire LOISON
Light and Matter Institutehttp://ilm.univ-lyon1.fr/

Theoretical Physical Chemistry Group



De : Josef Melcr notifications@github.com
Envoyé : mardi 6 septembre 2016 17:58
À : NMRLipids/NmrLipidsCholXray
Cc : LOISON CLAIRE; Comment
Objet : Re: [NMRLipids/NmrLipidsCholXray] Reproducibility of Charmm results in original paper (#4)

It had struck me that this effect can be easily produced by a barostat if misconfigured or if it has some glitches or implementation inaccuracies... The Areas per lipid are different and they might be the source of the difference in the OPs.

Right now, I'm running pure POPC (Charmm36) with openMM7.1 and Gromacs5.1.2 to see, whether there are differences -- and I'll probably try exchanging the barostat in Gromacs to see whether there's an effect...

You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/4#issuecomment-244998892, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALF31Nvbi8Mxo8wmFi8UDr3SZIZHkD5uks5qnY27gaJpZM4D0kMT.

@ohsOllila
Copy link
Member Author

I think that the barostat issue is unlikely.
Claire probably refers to the dispersion correction. This is not used in CHARMM simulations (at least in the one I reported here) because it is not reported to be used in the original CHARMM36 parametrization work.

There is one thing to note about previous discussion:
I wrote in blog in response to Piggot (http://nmrlipids.blogspot.com/2016/07/nmrlipids-iii-preliminary-observations.html?showComment=1471859719162#c4198823452649742103) that:
"With Gromacs 5 there are some instructions how to set cut-off parameters to be combatible with CHARMM model:
http://www.gromacs.org/Documentation/Terminology/Force_Fields/CHARMM
However, I think that this does not fix the problem."
When writing this I did remember wrongly that we would have used these settings in NMRlipids I. However, I did check it now and it seemed that everything there had been done with Gromacs 4.5. where verlet cut-off scheme was not available, including the data I reported above and the one in the current NMRlipids III manuscript. For some reason, I had a feeling that Gromacs 5 instructions do not fix the issue. I do not know why since I cannot find any test runs from my computer. Anyway, this (i.e. run simulation with Gromacs 5 according to the instructions) is what we should test first now, and it seems that J. Melcr had the Gromacs 5 simulation running already.

@jmelcr
Copy link
Contributor

jmelcr commented Sep 7, 2016

Anyway, this (i.e. run simulation with Gromacs 5 according to the instructions) is what we should test first now, and it seems that J. Melcr had the Gromacs 5 simulation running already.

Exactly, pure POPC running now with Charmm36. the mentioned Gromacs5 setting for C36 is used exactly (btw. standard output of Charmm-gui agrees with it).

@ohsOllila
Copy link
Member Author

Matti Javanainen also uploaded yesterday some CHARMM36 data of POPC/cholesterol mixtures in Zenodo: http://dx.doi.org/10.5281/zenodo.61649

I ran the analysis for pure POPC data (analysis for systems with cholesterol yet to be done) and added the results in figures:
https://github.com/NMRLipids/NmrLipidsCholXray/blob/master/FIGS/OrderParametersCHOL-eps-converted-to.pdf
https://github.com/NMRLipids/NmrLipidsCholXray/blob/master/FIGS/FormFactors-eps-converted-to.pdf
The order parameters and significantly closer to experiments but could be better. In form factor there is not significant improvement. Based on the parametrization paper and some other literature, I would expect better agreement for CHARMM36 with experiments.

@ClaireLoison
Copy link

ClaireLoison commented Sep 8, 2016

Concerning the reproducibility of the original C36 results (Klauda 2010)
for pure DPPC using NAMD. I have done some comparison of head and
tail SCD for different NAMD settings.

Despite small differences, I consider that the NPAT SCD results at 64AA /lipid in Figure 3 (bottom) of Klauda's article are very close to ours. Note that our settings are slightly
different, and we use NAMD (10-12 LJ cutoffs and 2fs timestep) whereas Klauda et al. used CHARMM (8-12 A LJ cutoffs and 1fs timestep).

orders_dppc_323k_npt_vs_npat-heads

orders_dppc_323k_npt_vs_npat-tails

On the opposite, forgetting the vdwForceSwitching and simulating in NPT
ensemble in which the area fluctuate to get a zero surface tension change the tail SCD.

ps: The impact of LJ cutoffs and timestep is also discussed in section 3.1.4 of this article :
Venable RM, Brown FLH, Pastor RW. Mechanical properties of lipid bilayers from molecular dynamics simulation. Chemistry and Physics of Lipids. 2015;192:60-74. doi:10.1016/j.chemphyslip.2015.07.014.

@jmelcr
Copy link
Contributor

jmelcr commented Sep 9, 2016

Nice plot, thank you!
What does "no vdwFS" mean -- what is used instead of force switching? Are there any changes in the cutoffs? This parameter looks like it gives the bit painful over-ordering effect, so I'm interested, what changes of the interaction potential (i.e. cutoff and its switches) make such change.

@ohsOllila
Copy link
Member Author

Where is the experimental data by Douliez actually taken from? By looking the table 1 in http://dx.doi.org/10.1016%2FS0006-3495(95)80350-4, it seems to me that the values at T=323K would be smaller than in the current plot.

@ClaireLoison
Copy link

For the (no vdwFS ) simulation, the NAMD

parameter vdwForceSwitching is turned off.

This parameters changes the way the switch of LJ interactions

between the two cutoffs is performed.

As far as I understood, the default NAMD

is a LJ energy switch between the two cutoffs,

but the force is not continuous. In CHARMM, both

LJ energy and forces are continuous at both cutoffs.

Here is the explanation form NAMD UG :

"If both switching and vdwForceSwitching are set to on, then CHARMM force
switching is used for van der Waals forces. "

see also

http://www.ks.uiuc.edu/Research/namd/mailing_list/namd-l.2011-2012/2609.html

The reason for studying this is twice for us :

  • we had done some simulations forgetting about vdwForceSwitching !

Since the area per lipid is different with different switching functions,

it is interesting so investigate the impact of different area per lipid on results.

  • we want to reproduce the NaCL/lipid NBFIX from Roux et al, and have some difficulties with that.

    As far as I understood given the data the authors have given me,

they also had vdwForceSwitching? turned off for their optimisation of the parameters.


De : Josef Melcr notifications@github.com
Envoyé : vendredi 9 septembre 2016 08:48
À : NMRLipids/NmrLipidsCholXray
Cc : LOISON CLAIRE; Comment
Objet : Re: [NMRLipids/NmrLipidsCholXray] Reproducibility of Charmm results in original paper (#4)

Nice plot, thank you!
What does "no vdwFS" mean -- what is used instead of force switching? Are there any changes in the cutoffs? This parameter looks like it gives the bit painful over-ordering effect, so I'm interested, what changes of the interaction potential (i.e. cutoff and its switches) make such change.

You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/4#issuecomment-245833738, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALF31EBMwC_qvU0uVeibxRzcwycLiJPVks5qoQFUgaJpZM4D0kMT.

@ClaireLoison
Copy link

Hi, thanks for your attentive remark. For Douliez et al. I have taken the data directly from the article.

Douliez JP, Léonard A, Dufourc EJ. Restatement of order parameters in biomembranes: calculation of C-C bond order parameters from C-D quadrupolar splittings. Biophysical journal. 1995;68(5):1727-1739. doi:10.1016/S0006-3495(95)80350-4.

Except for mistakes, I took the value of Table 3 at 50 degres.

Another remark considering this article :

maybe one could also try to see whether the even-odd

effect is obtained in the simulations at various temperatures ?

SCC is as easy as the SCD to calculate from simulations.

Since the even-odd effect correspond to the derivative of the SCC

It seems even more difficult to reproduce than the general trend of SCD.

Even more at various temperatures...

?


De : ohsOllila notifications@github.com
Envoyé : vendredi 9 septembre 2016 10:01
À : NMRLipids/NmrLipidsCholXray
Cc : LOISON CLAIRE; Comment
Objet : Re: [NMRLipids/NmrLipidsCholXray] Reproducibility of Charmm results in original paper (#4)

Where is the experimental data by Douliez actually taken from? By looking the table 1 in http://dx.doi.org/10.1016%2FS0006-3495(95)80350-4, it seems to me that the values at T=323K would be smaller than in the current plot.

You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/4#issuecomment-245846478, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALF31OMELOQRyaaTqidwZQEhtbfqJmfhks5qoRJNgaJpZM4D0kMT.

@ohsOllila
Copy link
Member Author

Ok. My mistake. I was looking the table from wrong lipid.

@ClaireLoison
Copy link

ClaireLoison commented Sep 9, 2016

Here are some results for SCD of pure POPC tails at 303K, calculated using NAMD. 72 lipids and about 2000 waters, for 100ns NPT. The SCD of the tails reproduce well the C36 data by Klauda et al., 2010 obtained with CHARMM, but both sn-1 and sn-2 order parameters are higher than the experimental data. Maybe this is due to a too small area/lipid in the NPT simulation ? For the NAMD results, the area per lipid is 64.8 AA, relatively close to the exp. one.
orders_popc_tails

@ohsOllila
Copy link
Member Author

Thanks! Could you share also the numerical values so that we can put the Gromacs results in the same plot? The numerical values from Gromacs are here:
https://github.com/NMRLipids/MATCH/tree/master/Data/Lipid_Bilayers/POPC/T310K/MODEL_CHARMM36
The temperature is slightly higher but it would be still worth of putting in the same figure.

@ClaireLoison
Copy link

Here is the data for our NAMD-C36 results : the values are followed by their variance, the error estimate

is about sqrt(variance)/26.


72 POPC + 2200 waters

NPT, 303K, 1atm, Lx/Ly fixed

NAMD

beta1 -3 0.07640677098708852 0.1747629068218557

beta2 -3 0.07508321598444838 0.17484878758022854

alpha1 -2 -0.03807238141291847 0.20956378114476149

alpha2 -2 -0.0257067386401897 0.20767275880799413

g31 -1 0.2388885768775786 0.08607526591923258

g32 -1 0.2530631511040515 0.08004482593269276

g2 0 0.20660125026805878 0.09925535984770309

g11 1 0.18019627359407536 0.12318823224085698

g12 1 0.04018602628337821 0.14488592088378308

sn2sn1 2 0.22525701866525205 0.044788047891244534 0.0913459913206161 0.06298796645580687

sn2sn1 3 0.17072383587417905 0.05983885256685818 0.20537798031009685 0.054061884452644586

sn2sn1 4 0.20233165522851435 0.05475314491851802 0.20034825282091387 0.05440962814752503

sn2sn1 5 0.20231784487335827 0.054147959989756224 0.20976385371144332 0.052973475529369055

sn2sn1 6 0.21541941789042587 0.05201180330379692 0.18644196996350526 0.05766554736771268

sn2sn1 7 0.20515775030881273 0.05377655463402382 0.17649410132895962 0.05985269125024771

sn2sn1 8 0.20555136158534454 0.05382470579698103 0.09802525883090739 0.07196403989847149

sn2sn1 9 0.18762370378771778 0.05709155452568148 0.05536004669025257 0.18209310216161567

sn2sn1 10 0.1799601505917554 0.0581497470636555 0.044096302426066006 0.1851044843406602

sn2sn1 11 0.16026646313264598 0.06125112428528533 0.07007914618895074 0.0762407815353155

sn2sn1 12 0.14813536524386606 0.0629708392759908 0.1142848323421472 0.06684090802165985

sn2sn1 13 0.12713875204681396 0.06532279602467393 0.12123498946244926 0.0656729125376915

sn2sn1 14 0.11131470465324475 0.06677228205642875 0.1263016863989463 0.06497669721425287

sn2sn1 15 0.08248560786365253 0.06862334500419658 0.11467703440384294 0.06636734644952041

sn2sn1 16 0.0 0.0 0.10380040692872766 0.06699027824107469

sn2sn1 17 0.0 0.0 0.08121245520460374 0.06855480483810632

Here is the CHARMM simulation data I extracted from the paper by Klauda et al.

using the figure 9 and http://arohatgi.info/WebPlotDigitizer/app/


POPC from the C36 paper,

DOI: 10.1021/jp101759q

figure 9 , upper pictures

green circles (sn-2)

  1. 0.09286
  2. 0.1022
  3. 0.2015
  4. 0.2000
  5. 0.2065
  6. 0.1841
  7. 0.1715
  8. 0.1048
  9. 0.06560
  10. 0.03807

11.0 0.07207

12.0 0.1145

13.0 0.1200

14.0 0.1214

15.0 0.1102

16.0 0.1008

17.0 0.08027?


POPC from the C36 paper,

DOI: 10.1021/jp101759q

figure 9 , upper pictures

blue diamonds (sn-1)

  1. 0.2071
  2. 0.1632
  3. 0.1958
  4. 0.1986
  5. 0.2135
  6. 0.2074
  7. 0.2102
  8. 0.1952
  9. 0.1872
  10. 0.1667
  11. 0.1536
  12. 0.1331
  13. 0.1144
  14. 0.08406


Claire LOISON
Light and Matter Institutehttp://ilm.univ-lyon1.fr/

Theoretical Physical Chemistry Group



De : ohsOllila notifications@github.com
Envoyé : vendredi 9 septembre 2016 13:29
À : NMRLipids/NmrLipidsCholXray
Cc : LOISON CLAIRE; Comment
Objet : Re: [NMRLipids/NmrLipidsCholXray] Reproducibility of Charmm results in original paper (#4)

Thanks! Could you share also the numerical values so that we can put the Gromacs results in the same plot? The numerical values from Gromacs are here:
https://github.com/NMRLipids/MATCH/tree/master/Data/Lipid_Bilayers/POPC/T310K/MODEL_CHARMM36
The temperature is slightly higher but it would be still worth of putting in the same figure.

You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/4#issuecomment-245888255, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALF31Bhz_T0bEzhbcEz-euOZByhTSsiGks5qoUMMgaJpZM4D0kMT.

@jmelcr
Copy link
Contributor

jmelcr commented Sep 12, 2016

Great, I'll bring up sim-data from openMM and Gmx512 later today.

@jmelcr
Copy link
Contributor

jmelcr commented Sep 12, 2016

I ran simulations based on optimal parameters and simulation setup from charmm-gui paper and gromacs site (agreeing together).
System consists of POPC membrane and water.
The compared software used was:
Gromacs 5.1.2 and openMM 7 (openMM is used in Charmm sim package for GPU accelerated runs).
It looks that the parameters used for the older simulation with Gromacs 4.5, as stated in NMRlipids1 paper, contain some small glitches, or the software (Gmx4.5) or its setup was systematically biased towards more rigid Charmm36 membranes.
Either way, current simulations (100ns both, but elongation doesn't change the result) show that at least Gromacs and openMM are consistent together. (see picture)
dops_openmm_gmx_compar

@jmelcr
Copy link
Contributor

jmelcr commented Sep 12, 2016

Updated the figure - added data from Claire's post (NAMD simulation and Charmm36 paper data), only inverted the values.

dops_openmm_gmx_compar
XmGrace file:
DOPs_openMM_GMX_compar.agr.zip

According to the obtained values, there is close to no difference in between current simulation softwares -- no secret hidden magic in obscure simulation settings and VdW cutoffs. Once the simulations use same setting, the results begin to match within some (in)accuracy -- as they should. Howgh.

I'll upload the obtained trajectories to Zenodo for all of you to check and use. In addition, we will have a set of trajectories of the same system, which were generated by 3 different simulation engines and results that match well.
Claire, could you also upload your trajectories of POPC to Zenodo, please?

@mattijavanainen
Copy link

mattijavanainen commented Sep 12, 2016

The curves look more similar than the ones in the Charmm-GUI paper (DOI: 10.1021/acs.jctc.5b00935), where the problem with pre-5.0 era Gromacs simulations with Charmm36 is also documented. I wonder what's the reason behind the better agreement observed here. Any ideas?

I provided links to some Charmm36 POPC trajectories with and without cholesterol in the blog (the ones Samuli refers to above); perhaps, using this data, we should check whether Gromacs 5.0 and Gromacs 5.1 give identical results.

Anyway, we should clearly stop using any Charmm36 data produced with pre-5.0 era Gromacs.

@jmelcr
Copy link
Contributor

jmelcr commented Sep 13, 2016

They claim they used Gromacs 5.0 in the Charmm-GUI paper. In release notes to version 5.0.2 there is:

Release notes for 5.0.2

Fixed major bug that affects simulations that:
    use versions 5.0 or 5.0.1 and
    use GPU acceleration and
    use PME electrostatics and
    were run with the default mdrun option -tunepme, and did actually change the electrostatics cut-off as a result of the PP-PME tuning.
The bug means the effective LJ cutoff is the same as the tuned electrostatics cutoff, which will break most model physics.

The bug does not affect simulations that:
    use only CPUs or
    use LJ-PME or
    use mdrun -notunepme or did not change the electrostatics cut-off as a result of the PP-PME tuning.
...
Simulations using vdwtype force-switch are affected, but less severely because of the shape of the force curve as it rises from zero after the intended cutoff distance.

So it also depends what settings was used in the paper (non-GPU/GPU). I didn't find it in the Charmm-GUI paper.
Gromacs 5.1.1 already has all fixes from 5.0.7. This might be the reason that the results agree better.
It's interesting that they discuss potential-based swithing in the Charmm-GUI paper, although it is discouraged in Gromacs manual v5.0:

Potential-switch
Smoothly switches the potential to zero between rvdw-switch and rvdw.  
Note that this introduces articifically large forces in the switching region 
and is much more expensive to calculate. 
**This option should only be used if the force field you are using requires this.**

The differnce between force/potential-based switching can also be expected, since they change the potential form of the classical model -> interactions are different -> ensembles are different. And this difference is pronounced in aggregates, in which VdW dispersion plays a large role -- lipid bilayers.
When you look at Figure 2 from the Charmm-gui paper, which compares APLs and DOPs, Gromacs with force-based switch is already quite OK.

Matti:
Anyway, we should clearly stop using any Charmm36 data produced with pre-5.0 era Gromacs.

Yes, use the best available software for the task =]
But why stay with Gromacs when there's no suspicion in NAMD and openMM for Charmm36? After all, they all give matching results -- and now we have evidence for the paper that they do.

perhaps, using this data, we should check whether Gromacs 5.0 and Gromacs 5.1 give identical results.

My guess is that it would be the easiest for you to compare -- significant differences should be visible after short runs. Also depends what 5.0.x version you used and whether it included this tunepme - or some other bug =]
We can add this check to SI -- it looks to me that we will have to touch this Charmm36-software issue somewhere in the article at least with a short paragraph in SI.

@jmelcr
Copy link
Contributor

jmelcr commented Sep 13, 2016

Here are the trajectories (POPC, 303K, Charmm36, 200ns):
openMM 7 : DOI
Gromacs 5.1.2 : DOI

@mattijavanainen
Copy link

Well, now we have both Gromacs 5.1.2 and Gromacs 5.0.7 trajectories available for comparison, Samuli could you take a look at this?

My arguments for staying in Gromacs (>5.0) are:

  1. The correct Charmm36 LJ options are available, unlike in openMM.
  2. It's so much faster than anything else, including NAMD.
  3. all other FF's are simulated with it.
  4. This would force us to understand why the results are not identical :)

@ohsOllila
Copy link
Member Author

I will take a closer look at this, but I am currently pretty busy with applications. Based on this discussion it seems to me that with Gromacs 5 with correct cut-off treatment we get results in reasonable agreement with NAMD and literature results. For NMRlipids III this is enough and we already have the simulations of POPC cholesterol mixtures with Gromacs 5 waiting in Zenodo for the analysis. About 5.1.2 vs. 5.0.7 comparison, we indeed have both trajectories but I think that they are with little bit different temperatures. Anyway, based on general picture it seems that the results are consistent enough for NMRlipids III.

Just for curiosity, how large is the performace difference between NAMD, openMM and Gromacs?

@mattijavanainen
Copy link

mattijavanainen commented Sep 13, 2016

On our desktops Gromacs is ~3–5(!) times faster than NAMD depending on whether GPU is used. For this test we used the files as provided by Charmm-GUI. This drastic difference in performance is also evident here: https://research.csc.fi/sisu-scalability-tests

I have no experience with openMM. However in my opinion, due to the lack of the correct LJ modifiers (this is stated in the Charmm-GUI paper), openMM should not really be considered for the Charmm36 simulations, just like Gromacs versions <5.0.

@TomPiggot
Copy link

Hi,

The order parameters are too high for the GROMACS 4.5 simulations due to the use of potential rather than force switching. This can be seen in both the above NAMD simulations where the no vdwFS simulation uses potential switching (it is probably also worth noting that this simulation is in agreement with a previous NAMD simulation I did with this setting; see table S3 of http://pubs.acs.org/doi/suppl/10.1021/ct3003157) and also, as already mentioned, seen in the simulations reported by Lee et al. using both force and potential switching in GROMACS 5. If you actually wanted to get force switching in GROMACS 4.5 you would have needed to have used a shift cut-off. I've had a look at the mdp's for DPPC/POPC and they don't use this. I didn't realise when we published our work that a shift cut-off could achieve force switching in GROMACS.

As for the order parameters in GROMACS 5, it is very interesting that Joseph's simulations show close agreement between GROMACS 5 and other softwares in contrast to Lee et al. who claimed that the membranes were consistently slightly too ordered. I guess we could also run a simulation using GROMACS 5.0 as per Lee et al. (with/without GPU) to test Joseph's hypothesis (and seeing if the cut-off is changed). That said, I imagine it would probably be easier to contact the corresponding author of the paper and ask if they can look at the log files to see if this was the case (and to also confirm the details, such as did they use 5.0 or another of the 5.0.x series). In any case, I've just set running some DPPC/POPC membranes in 5.0.6 to confirm what Joseph's simulations show, so we can at least be sure that the problem is something with the Lee et al. simulations.

Cheers

Tom

@mattijavanainen
Copy link

mattijavanainen commented Sep 14, 2016

Tom, if you did not notice, I already posted some POPC simulations run with Gromacs 5.0.7 to the blog some time ago, see: https://doi.org/10.5281/zenodo.61649

@TomPiggot
Copy link

Great, thanks. I hadn't noticed. I've actually found a GROMACS 5.1.2 simulation I had for POPC too (500 ns, 0.8/1.2 nm cut-off). I'm just having a look at the order parameters, etc. now.

@ohsOllila
Copy link
Member Author

It is also good to note that the numerical results from Matti's simulation are here:
https://github.com/NMRLipids/MATCH/tree/master/Data/Lipid_Bilayers/POPC/T310K/MODEL_CHARMM36

@TomPiggot
Copy link

TomPiggot commented Sep 14, 2016

So despite the simulations of both Matti and Joseph having slightly more disordered membranes (making GROMACS in closer agreement with other softwares), the simulation I mentioned above performed using GROMACS 5.1.2 is in better agreement with the results of Lee et al. The differences aren't huge but then again the differences reported by Lee et al (using the force switching) weren't huge either. I should also mention again that this simulation is 500 ns in length and is using a 0.8/1.2 force switch cut-off (which should likely make the membrane slightly more disordered than using 1.0/1.2 as was used in the simulations of Joseph/Matti).

The area per lipid from the simulation is shown in the plot below (the average is 0.640 nm^2 using the whole trajectory), matching closely to the Lee et al. GROMACS/POPC results of 0.641 (albeit with a 1.0/1.2 nm cut-off).

apl

Below are the order parameters for the sn-1 chain (as calculated using the MATCH scripts and I also checked this with a VMD tcl script (calc_op.tcl) that averages over the hydrogens; Claire I guess you probably used this for your NAMD sims?):

label Order_Parameter_1 Order_Parameter_2
2 -0.237379 -0.205151
3 -0.180745 -0.165733
4 -0.208016 -0.20509
5 -0.208754 -0.20537
6 -0.221026 -0.219797
7 -0.211828 -0.211809
8 -0.211395 -0.212408
9 -0.196605 -0.196309
10 -0.189479 -0.189202
11 -0.170516 -0.169722
12 -0.159337 -0.157645
13 -0.137763 -0.13638
14 -0.120842 -0.119277
15 -0.0910891 -0.0896848

And plotted against Joseph's values reported from his GROMACS simulation:

deutop_tp_jm

And also plotted against Claire's NAMD simulations (for both sn-1 and sn-2 chains here):

deutop_sn12_tp_cl

While the differences aren't huge, there are consistent differences in the order parameters. The differences between GROMACS and NAMD are also of a similar order and nature to those reported by Lee et al. (the POPC order parameter plots are in the SI of the paper, although it is hard to work out the exact numbers given the large scales and symbols used in these plots), who also said the increases were consistent but not significant.

As to why I am seeing differences to Joseph and Matti, I am not sure. I am running a series of simulations/tests to examine starting structure, cut-off, where the force field/lipid parameters are from and simulation length so as to hopefully figure this out. I should also say there are repeats of everything running too, so as we can hopefully be confident with the results. I will report back and can also share these simulations once completed.

The differences really aren't huge anyway but it still would be nice to work out why this consistent small increase in order of the membranes is (well, if it is!) happening. That said, I really don't think it is anything to get overly concerned about.

@jmelcr
Copy link
Contributor

jmelcr commented Sep 15, 2016

@TomPiggot

0.8/1.2 force switch cut-off (which should likely make the membrane slightly more disordered than using 1.0/1.2 as was used in the simulations of Joseph/Matti)

I'm afraid this is not generally valid. In addition, one should use the cutoff scheme belonging to the Force field, otherwise the potential form is being changed (and hence the results).
I don't understand to motivation to twiddle with these parameters in the charmm-gui paper, they probably just did it to see what it provides...

from charmm-gui paper:
10–12 Å is consistent with the force-based switching range used to develop the remainder of the additive CHARMM FF

Bearing in mind that you have a modified potential form for LJ interaction (through different switching), the agreement of the reported curves is OK. Notice that in Fig 2 in charmm-gui paper the difference is larger than the one shown here.

Could you do the simulation with the same settings as we did? The results must match within convergence-precision.
FYI, I used GPU for my run.
And I think we've met in Spain on the beautiful GRC in Gerona, Hi, good to virtually see you again!

@mattijavanainen (choice of software)
In experiments, the results do not depend on the fluorometer or the microscope -- it would be sad if they did.
The same must work for simulation software (unless it is buggy).
The choice of the software depends on what you need to simulate. For example on my worksation, openMM runs 2-3x faster than Gromacs - it uses the GPU better in my setup. Luckily, at least in version 7, openMM can read Gromacs topology files directly, so I can interchange almost freely.
The force switch is also taken care of:

from charmm-gui paper:
Although the standard nonbonded force calculation in OpenMM only supports the potential-based switching function, we provide an implementation of the force-based switching function using OpenMM’s CustomNonbondedForce class in the scripts generated by CHARMM-GUI.

@TomPiggot
Copy link

TomPiggot commented Sep 15, 2016

Hi Josef (PS, sorry for the incorrect spelling of your name before!). Yes, good to chat again, although my memories of the conference are somewhat hazy given the free booze on offer!

As for the cut-off settings with CHARMM, I probably should have been a bit more explicit in my previous message. I completely agree with you that random changes to the cut-off for a force field should not be done. Indeed some of my previous work was the first (I think) to highlight how both the change from force to potential switch and also which variant of TIP3P water model you use, can both massively impact upon the membrane properties with CHARMM36. Indeed, both of these (and other) settings should be classified as much as part of the force field as any of the atomtypes, etc. However, the one thing with the CHARMM force fields that has been a little more variable over time is the point at which the van der Waals interactions are switched off. In the older CHARMM force fields this value was 0.8 nm and indeed this was the value primarily used in the original CHARMM36 lipid force field parameterisation of Klauda et al. (http://pubs.acs.org/doi/abs/10.1021/jp101759q). However, Klauda also tested using a switching point of 1.1 nm due to the fact that this had been frequently used in the NAMD software. This resulted in a slightly more ordered membrane (e.g. lower APL, higher order parameters, thicker membrane, etc.) as can be understood through inclusion of some more of the longer range, attractive, van der Waals interactions. Therefore, it could be argued that either one of these values could be valid for CHARMM36 membrane simulations but based upon the membrane properties the most reasonable value to use would be 0.8 nm. Moving forwards from the original parameterisation of the CHARMM36 lipids, I believe the newer CHARMM36 protein force field was subsequently parameterised with a point to begin switching off the van der Waals interactions of 1.0 nm (IIRC this is true for other more recent parts of the CHARMM force field too but please don't take my word as gospel on this without looking). This is why in the paper of Lee et al., they tested both the 0.8 and 1.0 nm switching points with a complete cut-off at 1.2 nm for the van der Waals. While the former actually in general provides better membrane properties (and, given the fact that it was used in the vast majority of the original CHARMM36 lipid parameterisation, should probably be used for pure CHARMM36 membrane simulations IMHO) they recommended the latter (i.e. 1.0/1.2) to be consistent with other parts of the CHARMM36 force field parameters. As to why they also tested 0.8/1.0 in GROMACS, I am not completely sure. Perhaps to try and optimise membrane properties for pure membrane simulations I guess.

Anyway, so when I previously talked about simulations using different cut-off's I was simply referring to simulations with 0.8/1.2 and 1.0/1.2 nm for the van der Waals interactions. Both of these are valid settings and, as I mentioned above, the former (IMHO) is more appropriate for pure membrane systems.

All that said, yes I also agree that simulations reproducing what has already been done by yourself and Matti is the easiest way to work out what is going on. This is why the vast majority of the simulations I have set going are using the 1.0/1.2 nm cut-off and indeed I have 4 simulations running using the 200 ns structure you provided as input (my runs are cpu only). However, given what is known from the different settings (0.8/1.2 versus 1.0/1.2) and the fact the GROMACS 5.1.2 simulation I had to hand used the 0.8/1.2 settings that should have resulted in lower order parameters than your simulations, not higher, I am not convinced that there is agreement between our simulations. The simulations that are now running will show whether this is the case or not!

Cheers

Tom

@mattijavanainen
Copy link

We should perhaps try to organize the results with different software, versions, cut offs, temperatures etc. in a table format as I find it quite hard to follow the results both here and in the blog.

@jmelcr: Good to hear that the proper LJ options are available, and that openMM performs so well on a GPU. Have you tried it on a CPU – or more importantly – on a supercomputer/cluster? How is the performance there?

Actually I am not sure whether all MD codes are expected to provide the exact same result as these depend heavily on implementation details, which are not bugs. Therefore the obvious question is whether a force field should be used on multiple MD codes at all, or whether one should stick to the one used for the parameterization. Or in other words, should the implementation of algorithms be considered to be a fixed part of the force field? Unfortunately our exercise with Charmm36 seems to support the former view.

@jmelcr
Copy link
Contributor

jmelcr commented Sep 29, 2016

FYI, I'm running now POPC+30%Chol (using files from Matti) to see, whether we get same/different results. #HashTagDoubleCheck =]

edit: results regarding DOPs match well.

ohsOllila added a commit to NMRLipids/MATCH that referenced this issue Sep 29, 2016
ohsOllila added a commit that referenced this issue Sep 29, 2016
@ohsOllila
Copy link
Member Author

I plotted the results from this discussion into a figure below.

It seems to me now that with CHARMM, NAMD, Gromacs 5.x and openMM we are able to get results for sn-1 chain which are not quite the same but still pretty close to each others. It seems that this accuracy is currently enough for NMRlipids III. Thus understanding the fine details and origins of these differences is not currently necessary for this project, however, it may relevant for some other reasons. The results from all program packages give slightly larger order parameters from CHARMM36 than the experimental values from Ferreira et al.

The results from Gromacs 4.5 are larger than from other programs. Based on this discussion and the literature origin of the reason for this seems quite clear. In conclusion, we should not use simulations from Gromacs 4.5 for CHARMM36 anymore.

There is, however, a new problem: The results for sn-2 chain from Gromacs 5.0. simulation (http://doi.org/10.5281/zenodo.61649) are significantly different than the reported results from other simulations (see figure below). Now we need to find out what is the reason for this and if we can use already available simulations from Gromacs 5.0 with cholesterol in NMRlipids III. I will double check the order parameter calculation. The simulation running by Melcr is useful for this as well.

@TomPiggot
Could you share the numerical values for sn-2 and the used temperature from your simulation with Gromacs 5.1.2 mentioned above (or did I miss these somehow)?

orderparametersprograms

@TomPiggot
Copy link

The temperature was 300 K in this one simulation. The sn-2 order parameters are:

label Order_Parameter_1 Order_Parameter_2
2 -0.0695387 -0.107803
3 -0.201165 -0.2102
4 -0.19998 -0.204861
5 -0.212082 -0.211266
6 -0.188432 -0.188378
7 -0.179365 -0.179644
8 -0.103709 -0.10394
9 -0.0596121 nan
10 -0.0466988 nan
11 -0.078607 -0.07864
12 -0.121692 -0.121487
13 -0.128393 -0.127714
14 -0.132578 -0.133192
15 -0.121265 -0.121411
16 -0.110133 -0.110297
17 -0.0861884 -0.0862828

The series of simulations I mentioned above have just recently finished so I will report back with the analysis soon on these.

ohsOllila added a commit to NMRLipids/MATCH that referenced this issue Oct 3, 2016
ohsOllila added a commit that referenced this issue Oct 3, 2016
@ohsOllila
Copy link
Member Author

jmelcr added a commit to jmelcr/NmrLipidsCholXray that referenced this issue Oct 4, 2016
* Add figure based on results at NMRLipids#4

* Add results by Tom Piggot from NMRLipids#4
@ohsOllila
Copy link
Member Author

ohsOllila commented Oct 4, 2016

I opened a new issue about the CHARMM36 results from Gromacs 5.0 (http://dx.doi.org/10.5281/zenodo.61649, see comment #4 (comment) above), because this does not seem to be a version issue: #9

@jmelcr
Copy link
Contributor

jmelcr commented Oct 20, 2016

Results for POC+30%Chol presented in issue #9.
They suggest that Charmm-gui generated parameters are unreliable as they are not consistent between openMM and Gromacs giving different results.
See the comments there for the discussion about this issue.

@TomPiggot
Copy link

Firstly apologies for my slowness on this, I've been rushed off my feet with work. The series of simulations I mentioned above (GROMACS 5.0.6, 4 x 500 ns POPC started from the 200 ns structure of Josef and also 2 x 500 ns POPC from a different starting structure and different CHARMM/GROMACS parameter conversion I already had) are in pretty close agreement with the reported results of Lee et al. in the CHARMM-GUI paper, i.e. that GROMACS does have a slightly more ordered membrane (lower APL, higher order parameters) than those reported for other softwares. Below are the APL stats (first 100 ns not included):

From Josef's structure (simulation version, mean and standard deviation as outputted by g_analyze)
v1 6.441123e-01 1.281008e-02
v2 6.449186e-01 1.259714e-02
v3 6.449283e-01 1.363638e-02
v4 6.431565e-01 1.308374e-02

From my structure
v1 6.402857e-01 1.308591e-02
v2 6.446700e-01 1.328366e-02

And all the results from the order parameters (again for the final 400 ns of the simulations), firstly the simulations started from Josef's 200 ns structure (sn-1 followed by sn-2)

v1
2 -0.23786 -0.206664
3 -0.179019 -0.166172
4 -0.205429 -0.204094
5 -0.206812 -0.204257
6 -0.217672 -0.216874
7 -0.208583 -0.208766
8 -0.208179 -0.208146
9 -0.192514 -0.193149
10 -0.184578 -0.186043
11 -0.166683 -0.16678
12 -0.155336 -0.155324
13 -0.134594 -0.134124
14 -0.117352 -0.117659
15 -0.0886344 -0.0890111

2 -0.069326 -0.10744
3 -0.199527 -0.207576
4 -0.198875 -0.201811
5 -0.21162 -0.208488
6 -0.188418 -0.187347
7 -0.1796 -0.178289
8 -0.103579 -0.102729
9 -0.0582149 nan
10 -0.047307 nan
11 -0.0758519 -0.0773273
12 -0.119493 -0.119226
13 -0.126672 -0.126348
14 -0.13155 -0.130772
15 -0.120046 -0.120481
16 -0.109399 -0.109155
17 -0.0848264 -0.0856723

v2
2 -0.234607 -0.199974
3 -0.17738 -0.16358
4 -0.204563 -0.202093
5 -0.205903 -0.20258
6 -0.216892 -0.21544
7 -0.207354 -0.205771
8 -0.206274 -0.205997
9 -0.19061 -0.190959
10 -0.183031 -0.18355
11 -0.164376 -0.164129
12 -0.152881 -0.1524
13 -0.132722 -0.131414
14 -0.115877 -0.115647
15 -0.086704 -0.0871526

2 -0.0702182 -0.108638
3 -0.201127 -0.208529
4 -0.200014 -0.202096
5 -0.211858 -0.209647
6 -0.187754 -0.185957
7 -0.178492 -0.177316
8 -0.101 -0.102454
9 -0.0569779 nan
10 -0.0463444 nan
11 -0.0756132 -0.0764215
12 -0.117385 -0.116855
13 -0.123465 -0.123132
14 -0.12789 -0.127832
15 -0.116649 -0.116358
16 -0.106067 -0.106014
17 -0.0828203 -0.0828347

v3
2 -0.23457 -0.204076
3 -0.177333 -0.164188
4 -0.20359 -0.200898
5 -0.203154 -0.200846
6 -0.215603 -0.21367
7 -0.205997 -0.204585
8 -0.205011 -0.203909
9 -0.189155 -0.188407
10 -0.180953 -0.181169
11 -0.161295 -0.16273
12 -0.150285 -0.151245
13 -0.130135 -0.130172
14 -0.113321 -0.113886
15 -0.0854906 -0.0858524

2 -0.070868 -0.108084
3 -0.200327 -0.208288
4 -0.198537 -0.203306
5 -0.21004 -0.208719
6 -0.186624 -0.186489
7 -0.177214 -0.177901
8 -0.103052 -0.102381
9 -0.0589955 nan
10 -0.0463974 nan
11 -0.0761262 -0.0761639
12 -0.11768 -0.117583
13 -0.124658 -0.124026
14 -0.129407 -0.129061
15 -0.11816 -0.117579
16 -0.106998 -0.107076
17 -0.083193 -0.0841539

v4
2 -0.235879 -0.204568
3 -0.177277 -0.166203
4 -0.204776 -0.203097
5 -0.205691 -0.20368
6 -0.217741 -0.216337
7 -0.208518 -0.207813
8 -0.2083 -0.207004
9 -0.192546 -0.191328
10 -0.185277 -0.18455
11 -0.166986 -0.165056
12 -0.155527 -0.154184
13 -0.134194 -0.133942
14 -0.116864 -0.117268
15 -0.0880703 -0.0879237

2 -0.0742355 -0.107933
3 -0.203669 -0.209361
4 -0.201511 -0.204916
5 -0.212519 -0.212991
6 -0.189189 -0.189027
7 -0.179807 -0.180272
8 -0.102762 -0.103646
9 -0.0585025 nan
10 -0.0486904 nan
11 -0.0770694 -0.0785406
12 -0.11975 -0.119784
13 -0.126571 -0.126284
14 -0.131217 -0.130645
15 -0.119579 -0.119824
16 -0.109034 -0.108889
17 -0.085573 -0.0845909

And also from the simulations of the structure and force field parameters I already had:

v1
2 -0.240804 -0.209363
3 -0.181034 -0.166719
4 -0.207774 -0.204979
5 -0.207623 -0.205307
6 -0.219684 -0.218501
7 -0.21048 -0.209307
8 -0.209724 -0.209574
9 -0.193925 -0.193798
10 -0.185536 -0.186259
11 -0.16646 -0.166217
12 -0.154994 -0.154478
13 -0.133632 -0.13413
14 -0.117574 -0.117523
15 -0.0886182 -0.088447

2 -0.0743307 -0.1071
3 -0.203612 -0.209389
4 -0.200802 -0.203447
5 -0.212222 -0.210357
6 -0.188364 -0.187308
7 -0.179359 -0.178431
8 -0.102052 -0.103325
9 -0.0580626 nan
10 -0.0487744 nan
11 -0.079512 -0.0785068
12 -0.121181 -0.121671
13 -0.128274 -0.128056
14 -0.132918 -0.132941
15 -0.1211 -0.121267
16 -0.110101 -0.109592
17 -0.0851631 -0.0858753

v2
2 -0.238288 -0.203738
3 -0.178914 -0.165166
4 -0.206062 -0.203254
5 -0.206118 -0.202927
6 -0.218271 -0.215424
7 -0.209395 -0.206799
8 -0.208555 -0.207215
9 -0.192903 -0.191908
10 -0.185669 -0.184963
11 -0.167556 -0.166457
12 -0.155461 -0.155111
13 -0.134566 -0.135494
14 -0.118124 -0.118149
15 -0.0896336 -0.0889128

2 -0.0727035 -0.107453
3 -0.20093 -0.207223
4 -0.199815 -0.202171
5 -0.210448 -0.207875
6 -0.187139 -0.184723
7 -0.177535 -0.176518
8 -0.10188 -0.100423
9 -0.0557897 nan
10 -0.0469518 nan
11 -0.077022 -0.078206
12 -0.118597 -0.118443
13 -0.125772 -0.124888
14 -0.130414 -0.130687
15 -0.119556 -0.119275
16 -0.108546 -0.108661
17 -0.0848062 -0.0848275

These results reported above (sorry for all the numbers) are from simulations performed using the 1.0/1.2 nm vdW cut-off's. The same set of simulations performed with 0.8/1.2 nm vdW cut-off's demonstrate how changing from a 1.0 nm to 0.8 nm switching point (to match the original values of Klauda et al.) can slightly decrease the order:

From Josef's structure
v1 6.482740e-01 1.420514e-02
v2 6.475623e-01 1.340831e-02
v3 6.493069e-01 1.297804e-02
v4 6.441159e-01 1.380892e-02

From my structure
v1 6.470214e-01 1.353289e-02
v2 6.478693e-01 1.320012e-02

Hopefully this puts some of the doubts around this issue to bed. Yes, there is a slight differences in the order of POPC membranes when using GROMACS 5 compared to reported values when simulating in other MD packages (e.g. NAMD) but they are fairly minor and also they can be reproduced with different membrane structures and different origins of the GROMACS force field parameters. As to why they occur, I am still unsure. I also think this demonstrates my point that, if you are simulating a membrane system only, you should use the 0.8/1.2 nm cut-off scheme for the van der Waals interactions.

I'll upload these simulations and analysis to Zenodo when I get the chance.

Finally it's worth noting that I ran some DPPC simulations too that also agree with these points above on matching with Lee et al. and that 0.8/1.2 nm switching is better in GROMACS (I didn't include these results in this post as there are quite enough numbers already!).

@TomPiggot
Copy link

The POPC simulation files are on Zenodo now:

https://doi.org/10.5281/zenodo.164206
https://doi.org/10.5281/zenodo.164203

The trajectories have been processed with trjconv -skip 10 to make the uploads a reasonable size but the analysis reported above was all on the original trajectories.

@ohsOllila
Copy link
Member Author

I think that these issues are nowadays known and discussed in several publications after this discussion. However, I am not really sure if origins of all differences are understood. Anyway, I will close this issue now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants