Observables table #244

dweindl · 2020-01-25T08:45:19Z

Adapt PEtab to use separate table for defining observables instead of embedding them inside the SBML model (see discussion in #201)

Closes #241, closes #236, closes #201

codecov-io · 2020-01-25T08:47:16Z

Codecov Report

Merging #244 into develop will decrease coverage by 5.47%.
The diff coverage is 49.49%.

@@             Coverage Diff             @@
##           develop     #244      +/-   ##
===========================================
- Coverage    77.86%   72.38%   -5.48%     
===========================================
  Files           20       22       +2     
  Lines         1468     1644     +176     
  Branches       317      356      +39     
===========================================
+ Hits          1143     1190      +47     
- Misses         256      377     +121     
- Partials        69       77       +8

Impacted Files	Coverage Δ
petab/version.py	`100% <ø> (ø)`	⬆️
petab/composite_problem.py	`45.83% <ø> (ø)`	⬆️
petab/format_version.py	`100% <ø> (ø)`	⬆️
petab/visualize/data_overview.py	`100% <ø> (ø)`	⬆️
petab/migrations.py	`0% <0%> (ø)`
petab/yaml.py	`85% <100%> (+0.38%)`	⬆️
petab/C.py	`100% <100%> (ø)`	⬆️
petab/problem.py	`58.44% <14.28%> (-13.69%)`	⬇️
petab/observables.py	`31.57% <31.57%> (ø)`
petab/parameters.py	`82.82% <33.33%> (-1.55%)`	⬇️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c1cf32...f63ebe8. Read the comment docs.

No need to check for equal NOISE_DISTRIBUTION and OBSERVABLE_TRANSFORMATION anymore, so they are no longer included in the measurement table, and cannot differ for the same observableId

Closes #247 Was creating wrong observables before

yannikschaelte · 2020-01-28T18:05:54Z

.pylintrc

@@ -319,7 +319,7 @@ ignored-classes=optparse.Values,thread._local,_thread._local
 # (useful for modules/projects where namespaces are manipulated during runtime
 # and thus existing member attributes cannot be deduced by static analysis). It
 # supports qualified module names, as well as Unix pattern matching.
-ignored-modules=matplotlib
+ignored-modules=matplotlib,sympy,pandas,numpy


what was the problem?

pylint complained about correct imports from there.

yannikschaelte · 2020-01-28T18:07:38Z

.pylintrc

@@ -516,7 +516,7 @@ valid-metaclass-classmethod-first-arg=cls
 max-args=5

 # Maximum number of attributes for a class (see R0902).
-max-attributes=7
+max-attributes=12


can one just disable such nonsense stuff?

wouldn't call it nonsense, but the specific settings maybe subject to discussion. but to answer your question: yes, one can

doc/documentation_data_format.md

yannikschaelte · 2020-01-28T18:12:00Z

doc/documentation_data_format.md

+  output parameters defined in the observation model. The model can define
+  observables (see below) containing place-holder parameters which can be
+  replaced by condition-specific dynamic or constant parameters. Placeholder
+  parameters must be named `observableParameter${n}_${observableId}`


do we want to keep that for the moment? or use the a=1;b=2 thing?

I think decision is still pending. All changes still follow this format.

yannikschaelte · 2020-01-28T18:12:21Z

doc/documentation_data_format.md

-  All placeholders defined in the model must be overwritten here. If there are
-  not placeholders in the model, this column may be omitted.
+  All placeholders defined in the observation model must be overwritten here.
+  If there are not placeholders in the model, this column may be omitted.


Suggested change

If there are not placeholders in the model, this column may be omitted.

If there are no placeholders in the model, this column may be omitted.

yannikschaelte · 2020-01-28T18:23:54Z

petab/core.py

@@ -27,6 +26,17 @@ def get_simulation_df(simulation_file: str) -> pd.DataFrame:
    return pd.read_csv(simulation_file, sep="\t", index_col=None)


+def write_simulation_df(df: pd.DataFrame, filename: str) -> None:


looks pretty generic

yannikschaelte · 2020-01-28T18:24:29Z

petab/core.py

@@ -105,6 +126,8 @@ def get_observable_id(parameter_id: str) -> str:
    Returns:
        Observable ID
    """
+    warn("This function will be removed in future releases.",


... because that is not petab's topic, right?

This was based on the naming convention that we just kicked out. So no need to keep it, but let's phase it out slowly. I think I used it in the migration function to update models of the old format.

yannikschaelte · 2020-01-28T18:26:07Z

petab/core.py

@@ -250,7 +266,8 @@ def concat_tables(
        if isinstance(tmp_df, str):
            tmp_df = file_parser(tmp_df)

-        df = df.append(tmp_df, sort=False, ignore_index=True)
+        df = df.append(tmp_df, sort=False,


add comment what it does? never used a rangeindex.

Pretty sure you used it. That's the default index, if you don't set it yourself. In case we have a proper index we want to keep it, if not, we don't want two rows labeled "0".

yannikschaelte · 2020-01-28T18:28:04Z

petab/lint.py

    """
-    Check whether there are not multiple noise distributions for an
-    observable, and that the names are correct.
+    Ensure that noise distributions and observable transformations


observable transformations for observables :)

Maybe this way the link to the observableTransformation column is clearer... :) I see your point.

yannikschaelte · 2020-01-28T18:29:43Z

petab/measurements.py

@@ -47,6 +66,9 @@ def get_noise_distributions(measurement_df: pd.DataFrame) -> dict:
    Returns:
        Dictionary with `observableId` => `cost definition`
    """
+    warn("This function will be removed in future releases.",


why not just remove?

Still used in the script to update assignmentrule-style models. Would kick out in 0.2.0.

yannikschaelte

looks like it might make sense. with a grain of trust.

yannikschaelte · 2020-01-28T18:31:20Z

petab/parameter_mapping.py

@@ -476,7 +480,7 @@ def handle_missing_overrides(mapping_par_opt_to_par_sim: ParMappingDict,
            mapping_par_opt_to_par_sim[key] = np.nan
            _missed_vals.append(key)

-    if len(_missed_vals) and warn:
+    if not _missed_vals and warn:


is that right?

hm... seems not to be case...

yannikschaelte · 2020-01-28T18:34:14Z

petab/sbml.py

@@ -278,6 +287,9 @@ def add_model_output_with_sigma(
        observable_name:
            Any name
    """
+    warn("This function will be removed in future releases.",


just remove all this stuff?

yannikschaelte · 2020-01-28T18:35:08Z

petab/version.py

@@ -1 +1,2 @@
+"""PEtab version"""


setup.py might read in this file (didn't check), in that case automatic versioning might break.

it thought it evals it. will double check. was to satisfy pylint.

exec(read(version_file)), so it's safe

yannikschaelte · 2020-01-28T18:35:36Z

petab/visualize/helper_functions.py

@@ -479,12 +469,11 @@ def get_data_to_plot(vis_spec: pd.DataFrame,
        pre_cond = m_data[PREEQUILIBRATION_CONDITION_ID][ind_meas[0]]
        bool_preequ = (pre_cond == m_data[PREEQUILIBRATION_CONDITION_ID])
        # special handling is needed, if preequilibration cond is left empty
-        if (type(pre_cond) == np.float64) and np.isnan(pre_cond):
+        if isinstance(pre_cond, Number) and np.isnan(pre_cond):


yannikschaelte · 2020-01-28T18:37:25Z

tests/test_lint.py

+        OBSERVABLE_ID: ['0obsPar1noisePar',
+                        '2obsPar0noisePar'],
+        NOISE_PARAMETERS: ['', ''],
+        NOISE_DISTRIBUTION: ['', ''],


also test with that column non-existent?

Co-Authored-By: Yannik Schälte <31767307+yannikschaelte@users.noreply.github.com>

Release 0.1.0 Data format: * Introduce observables table instead of SBML assignment rules for defining observation model (#244) (moves observableTransformation and noiseModel from the measurement table to the observables table) * Allow initial concentrations / sizes in condition table (#238) * Fixes and clarifications in the format documentation * Changes in prior columns of the parameter table (#222) * Introduced separate version number of file format, this release being version 1 Library: * Adaptations to new file formats * Various bugfixes and clean-up, especially in visualization and validator * Parameter mapping changed to include all model parameters and not only those differing from the ones defined inside the SBML model * Introduced constants for all field names and string options, replacing most string literals in the code (#228) * Added unit tests and additional format validation steps * Optional parallelization of parameter mapping (#205) * Extended documentation (in-source and example Jupyter notebooks)

* Add pylint config * Fixes ys (#237) * fix merge error * add petablint yaml test * add parameters test * Parameter mapping should include all model parameters (#235) Closes #103 * Parameter mapping should include all model parameters * Known values should be filled in * Extend and update tests * Refactor parameter mapping * properly handle estimated and non-estimated parameters * Fix wrong parameter scale returned from mapping * ... * Allow initial concentrations / sizes in condition table (#238) * Allow species and compartments in condition table * Updated doc allowing for states etc in condition file * Update pylint: allow lower-case constants * Export __format_version__ * Fix returning floats as strings in case there are parameter names in the condition table Co-authored-by: LeonardSchmiester <leonard.schmiester@helmholtz-muenchen.de> * Barplots and Replicates with Simulation data (#214) Fixes #196, fixes #210, fixes #213 * Cleanup visualization (#240) * Add constants for visualization field IDs * .. and some others * start using them * formatting, ... * Observables table instead of SBML assignment rules (#244) Closes #201, closes #241 * Update data format doc for observable table * Add field name constants for observable table * Add observables table to petab.Problem * Update YAML schema and CompositeProblem * Add functions for writing PEtab dataframes to files * Deprecate SBML-observable functions * Implement validation for observable table * Add function for converting SBML-observable models to observable table * Use costants for PEtab table fields * Update PEtab files illustration * Fix most pylint issues (Closes #234) * Update vis to observalble table (Closes #246) No need to check for equal NOISE_DISTRIBUTION and OBSERVABLE_TRANSFORMATION anymore, so they are no longer included in the measurement table, and cannot differ for the same observableId * Fix and update flatten_timepoint_specific_output_overrides Closes #247 Was creating wrong observables before * Address review comments Co-authored-by: Yannik Schälte <31767307+yannikschaelte@users.noreply.github.com> * Release 0.1.0; file format version 1 * Fix parameter mapping: include output parameters not present in SBML model * Add convenience functions to petab.Problem * get_optimization_parameter_scales * get_optimization_to_simulation_scale_mapping * add tests * Fix petab/petab_schema.yaml missing in pypi package * Update pylint ignorelist * Update README * Remove obsolete functions ... related to hierarchical optimization, which should be kept outside PEtab * Let get_placeholders return an (ordered) list of placeholders because it is much more useful * Add check for valid identifiers (Closes #179) (#253) Co-authored-by: Polina Lakrisenko <p.lakrisenko@gmail.com> * Deprecate petab.problem.from_folder (Closes #245) ... as well as get_default_*_file_name * Release 0.1.1 * Barplot uniform coloring & yScale=log fix #196 (#255) * resolves #197 * small fix * change all barplot colors to blue * allow to extract only estimate parameters (#256) * allow to extract only estimate parameters * add docstrings * Visu callobs par (#262) * fix #261 * corrected flake8 error - line too long * deleted white space * add F403 to falke8 tests * Fix handling of numeric observable/noiseFormula in observable table (Fixes #264) * Add properties for fixed/free and scaled values (#268) * allow to extract only estimate parameters * add docstrings * return scaled versions of arrays * Update petab/problem.py Co-Authored-By: Daniel Weindl <dweindl@users.noreply.github.com> Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> * Observables function (#269) * allow to extract only estimate parameters * add docstrings * return scaled versions of arrays * Update petab/problem.py Co-Authored-By: Daniel Weindl <dweindl@users.noreply.github.com> * add get_observables function * add observables test; use observables file in petab test * fix typo * move get_observables to ..._ids * remove unused arg * add docstring * fix lint * fix pylint * fix flake8 Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> * Fix documentation hierarchy * Add functions to get all of fixed|free, scaled parameter values (#273) * allow to extract only estimate parameters * add docstrings * return scaled versions of arrays * Update petab/problem.py Co-Authored-By: Daniel Weindl <dweindl@users.noreply.github.com> * add get_observables function * add observables test; use observables file in petab test * fix typo * move get_observables to ..._ids * remove unused arg * add docstring * fix lint * fix pylint * fix flake8 * fix typo * streamline fixed|free|all, and scaled values * fix default args * fix codacy * address reviewer comment: return empty list * add docstring# * add more docstrings * fix var type error * address reviewer comments Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> * Default column to look for simulation results should be 'simulation' * PEtab COMBINE archives (#271) * Add create_combine_archive for generation of COMBINE archives * Add support for reading PEtab COMBINE archives * Add tests for COMBINE archive r/w * Fix sbml_observables_to_table - got broken in eb5453 * Increase test coverage (#278) * allow to extract only estimate parameters * add docstrings * return scaled versions of arrays * Update petab/problem.py Co-Authored-By: Daniel Weindl <dweindl@users.noreply.github.com> * add get_observables function * add observables test; use observables file in petab test * fix typo * move get_observables to ..._ids * remove unused arg * add docstring * fix lint * fix pylint * fix flake8 * fix typo * streamline fixed|free|all, and scaled values * fix default args * fix codacy * address reviewer comment: return empty list * add docstring# * add more docstrings * fix var type error * address reviewer comments * add parameter properties test * add tests for get/write_parameter_df * add measurements tests * add conditions tests * fix conditions create function * add parameter tests * add observables tests * fixup * add docstrings * address reviewer comments * random edit to see if codacy is happy * random stuff to annoy codecov Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> Co-authored-by: Yannik Schälte <31767307+yannikschaelte@users.noreply.github.com> Co-authored-by: LeonardSchmiester <leonard.schmiester@helmholtz-muenchen.de> Co-authored-by: Simon Merkt <49190262+MerktSimon@users.noreply.github.com> Co-authored-by: Polina Lakrisenko <p.lakrisenko@gmail.com> Co-authored-by: LaraFuhrmann <55209716+LaraFuhrmann@users.noreply.github.com>

dweindl added 2 commits January 25, 2020 09:43

Update data format doc for observable table

9ea913c

Add field name constants for observable table

b40bf80

dweindl added 3 commits January 25, 2020 10:09

Add observables table to petab.Problem

06c8879

Update YAML schema and CompositeProblem

69f0dee

Add functions for writing PEtab dataframes to files

98caae2

dweindl force-pushed the feature_241_obs_table branch from bf4f4bc to 98caae2 Compare January 25, 2020 12:15

dweindl added 3 commits January 25, 2020 14:29

Deprecate SBML-observable functions

891864b

Implement validation for observable table

d7b9380

Add function for converting SBML-observable models to observable table

1f150af

dweindl force-pushed the feature_241_obs_table branch from 6d19ab4 to 1f150af Compare January 25, 2020 15:24

dweindl added 2 commits January 25, 2020 17:18

Use costants for PEtab table fields

c157d9e

Update PEtab files illustration

fdf0246

dweindl requested a review from yannikschaelte January 25, 2020 16:19

dweindl changed the title ~~[WIP] Observables table~~ Observables table Jan 25, 2020

dweindl added 4 commits January 25, 2020 21:06

Various fixups

bed32bd

Fix most pylint issues (Closes #234)

df3d8d6

Update vis to observalble table (Closes #246)

c2ceadd

No need to check for equal NOISE_DISTRIBUTION and OBSERVABLE_TRANSFORMATION anymore, so they are no longer included in the measurement table, and cannot differ for the same observableId

Fix and update flatten_timepoint_specific_output_overrides

6f91515

Closes #247 Was creating wrong observables before

dweindl force-pushed the feature_241_obs_table branch from a5b8c4c to 6f91515 Compare January 26, 2020 15:10

dweindl mentioned this pull request Jan 27, 2020

Add script to update benchmark collection to observable table format LeonardSchmiester/Benchmark-Models#88

Merged

yannikschaelte requested a review from a team January 27, 2020 14:23

yannikschaelte reviewed Jan 28, 2020

View reviewed changes

yannikschaelte approved these changes Jan 28, 2020

View reviewed changes

dweindl and others added 2 commits January 28, 2020 20:19

Update doc/documentation_data_format.md

2bf2d36

Co-Authored-By: Yannik Schälte <31767307+yannikschaelte@users.noreply.github.com>

Address review comments

f63ebe8

dweindl merged commit c334edc into develop Jan 28, 2020

dweindl deleted the feature_241_obs_table branch January 28, 2020 19:36

dweindl mentioned this pull request Jan 28, 2020

Release 0.1.0 #248

Merged

dweindl mentioned this pull request Jan 29, 2020

Replace remaining string literals by constant where possible #236

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Observables table #244

Observables table #244

dweindl commented Jan 25, 2020 •

edited

Loading

codecov-io commented Jan 25, 2020 •

edited

Loading

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020 •

edited

Loading

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020

yannikschaelte Jan 28, 2020

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020 •

edited

Loading

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020

yannikschaelte left a comment

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020

dweindl Jan 28, 2020

yannikschaelte Jan 28, 2020

yannikschaelte Jan 28, 2020

dweindl Jan 28, 2020

	If there are not placeholders in the model, this column may be omitted.
	If there are no placeholders in the model, this column may be omitted.

		@@ -27,6 +26,17 @@ def get_simulation_df(simulation_file: str) -> pd.DataFrame:
		return pd.read_csv(simulation_file, sep="\t", index_col=None)


		def write_simulation_df(df: pd.DataFrame, filename: str) -> None:

Observables table #244

Observables table #244

Conversation

dweindl commented Jan 25, 2020 • edited Loading

codecov-io commented Jan 25, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dweindl Jan 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dweindl Jan 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yannikschaelte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dweindl commented Jan 25, 2020 •

edited

Loading

codecov-io commented Jan 25, 2020 •

edited

Loading

dweindl Jan 28, 2020 •

edited

Loading

dweindl Jan 28, 2020 •

edited

Loading