Skip to content

Commit

Permalink
Merge pull request #242 from glotaran/fix_perfomance
Browse files Browse the repository at this point in the history
Performance Fixes - full test suite down to 20 seconds! Good works guys!
  • Loading branch information
jsnel committed Nov 18, 2019
2 parents f68892b + d17bfe7 commit 3ca455d
Show file tree
Hide file tree
Showing 18 changed files with 493 additions and 210 deletions.
106 changes: 106 additions & 0 deletions .github/workflows/CI_CD_actions.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
name: Tests

on: [push, pull_request]

jobs:
lint:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v1
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8
- name: Lint with flake8
run: |
flake8 glotaran
docs:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v1
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install .
pip install -U -r requirements_dev.txt
- name: Show installed packages
run: pip freeze
- name: Build docs
run: |
make --directory=docs clean html
docs-links:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v1
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install .
pip install -U -r requirements_dev.txt
- name: Show installed packages
run: pip freeze
- name: Check doc links
run: |
make --directory=docs clean linkcheck
test:
runs-on: ${{ matrix.os }}
needs: [lint, docs]
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
python-version: [3.6, 3.7]

steps:
- uses: actions/checkout@v1
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements_dev.txt
- name: Show installed packages
run: pip freeze
- name: Run tests
run: |
py.test --cov=glotaran --cov-config .coveragerc -k 'not IrfDispersion' glotaran
deploy:
runs-on: [ubuntu-latest]
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
needs: test
steps:
- uses: actions/checkout@v1
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements_dev.txt
- name: Build dist
run: |
python setup.py sdist bdist_wheel
- name: Publish package
uses: pypa/gh-action-pypi-publish@master
with:
user: __token__
password: ${{ secrets.pypi_password }}
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ install:
script:
- pwd
- pip install -e .
- py.test --cov=glotaran --cov-config .coveragerc glotaran
- py.test -vv -s --cov=glotaran --cov-config .coveragerc glotaran
- pip freeze

after_success:
Expand Down
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@
pyGloTarAn is a python library for global and target analysis

[![latest release](https://pypip.in/version/glotaran/badge.svg)](https://pypi.org/project/glotaran/)
[![Build Status Linux + OsX](https://travis-ci.org/glotaran/pyglotaran.svg?branch=master)](https://travis-ci.org/glotaran/pyglotaran)
[![Build Status Windows](https://ci.appveyor.com/api/projects/status/76gkx10wyn2cd049?svg=true)](https://ci.appveyor.com/project/glotaran/pyglotaran)
[![Build Status](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fglotaran%2Fpyglotaran%2Fbadge&style=popout)](https://actions-badge.atrox.dev/glotaran/pyglotaran/goto)
[![Documentation Status](https://readthedocs.org/projects/glotaran/badge/?version=latest)](https://glotaran.readthedocs.io/en/latest/?badge=latest)
[![Coverage Status](https://coveralls.io/repos/github/glotaran/pyglotaran/badge.svg?branch=master)](https://coveralls.io/github/glotaran/pyglotaran?branch=master)

Expand Down
4 changes: 2 additions & 2 deletions docs/source/_templates/known_packages.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
For changes to take effect you might also have to run `make clean_all`
afterwards.
{% set known_packages=['glotaran.analysis', 'glotaran.builtin', 'glotaran.builtin.models', 'glotaran.builtin.models.doas', 'glotaran.builtin.models.kinetic_image', 'glotaran.builtin.models.kinetic_spectrum', 'glotaran.cli', 'glotaran.cli.commands', 'glotaran.examples', 'glotaran.io', 'glotaran.model', 'glotaran.parameter', 'glotaran.parse'] %}
{% set child_modules=['glotaran.analysis', 'glotaran.analysis.matrix_calculation', 'glotaran.analysis.nnls', 'glotaran.analysis.optimize', 'glotaran.analysis.parameter_server', 'glotaran.analysis.problem_bag', 'glotaran.analysis.residual_calculation', 'glotaran.analysis.result', 'glotaran.analysis.scheme', 'glotaran.analysis.simulation', 'glotaran.analysis.variable_projection', 'glotaran.builtin', 'glotaran.builtin.models', 'glotaran.builtin.models.doas', 'glotaran.builtin.models.doas.doas_matrix', 'glotaran.builtin.models.doas.doas_megacomplex', 'glotaran.builtin.models.doas.doas_model', 'glotaran.builtin.models.doas.doas_result', 'glotaran.builtin.models.doas.doas_spectral_matrix', 'glotaran.builtin.models.doas.oscillation', 'glotaran.builtin.models.kinetic_image', 'glotaran.builtin.models.kinetic_image.initial_concentration', 'glotaran.builtin.models.kinetic_image.irf', 'glotaran.builtin.models.kinetic_image.k_matrix', 'glotaran.builtin.models.kinetic_image.kinetic_image_dataset_descriptor', 'glotaran.builtin.models.kinetic_image.kinetic_image_matrix', 'glotaran.builtin.models.kinetic_image.kinetic_image_megacomplex', 'glotaran.builtin.models.kinetic_image.kinetic_image_model', 'glotaran.builtin.models.kinetic_image.kinetic_image_result', 'glotaran.builtin.models.kinetic_spectrum', 'glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_dataset_descriptor', 'glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_matrix', 'glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_model', 'glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_result', 'glotaran.builtin.models.kinetic_spectrum.spectral_constraints', 'glotaran.builtin.models.kinetic_spectrum.spectral_irf', 'glotaran.builtin.models.kinetic_spectrum.spectral_matrix', 'glotaran.builtin.models.kinetic_spectrum.spectral_relations', 'glotaran.builtin.models.kinetic_spectrum.spectral_shape', 'glotaran.cli', 'glotaran.cli.commands', 'glotaran.cli.commands.explore', 'glotaran.cli.commands.export', 'glotaran.cli.commands.optimize', 'glotaran.cli.commands.pluginlist', 'glotaran.cli.commands.print', 'glotaran.cli.commands.util', 'glotaran.cli.commands.validate', 'glotaran.cli.main', 'glotaran.examples', 'glotaran.examples.sequential', 'glotaran.io', 'glotaran.io.prepare_dataset', 'glotaran.io.reader', 'glotaran.model', 'glotaran.model.dataset_descriptor', 'glotaran.model.model', 'glotaran.model.model_attribute', 'glotaran.model.model_decorator', 'glotaran.model.model_property', 'glotaran.model.util', 'glotaran.parameter', 'glotaran.parameter.parameter', 'glotaran.parameter.parameter_group', 'glotaran.parse', 'glotaran.parse.parser', 'glotaran.parse.register'] %}
{% set known_packages=['glotaran.analysis', 'glotaran.builtin', 'glotaran.builtin.file_formats', 'glotaran.builtin.file_formats.ascii', 'glotaran.builtin.file_formats.sdt', 'glotaran.builtin.models', 'glotaran.builtin.models.doas', 'glotaran.builtin.models.kinetic_image', 'glotaran.builtin.models.kinetic_spectrum', 'glotaran.cli', 'glotaran.cli.commands', 'glotaran.examples', 'glotaran.io', 'glotaran.model', 'glotaran.parameter', 'glotaran.parse'] %}
{% set child_modules=['glotaran.analysis', 'glotaran.analysis.matrix_calculation', 'glotaran.analysis.nnls', 'glotaran.analysis.optimize', 'glotaran.analysis.problem_bag', 'glotaran.analysis.residual_calculation', 'glotaran.analysis.result', 'glotaran.analysis.scheme', 'glotaran.analysis.simulation', 'glotaran.analysis.variable_projection', 'glotaran.builtin', 'glotaran.builtin.file_formats', 'glotaran.builtin.file_formats.ascii', 'glotaran.builtin.file_formats.ascii.wavelength_time_explicit_file', 'glotaran.builtin.file_formats.sdt', 'glotaran.builtin.file_formats.sdt.sdt_file', 'glotaran.builtin.file_formats.sdt.sdt_file_reader', 'glotaran.builtin.models', 'glotaran.builtin.models.doas', 'glotaran.builtin.models.doas.doas_matrix', 'glotaran.builtin.models.doas.doas_megacomplex', 'glotaran.builtin.models.doas.doas_model', 'glotaran.builtin.models.doas.doas_result', 'glotaran.builtin.models.doas.doas_spectral_matrix', 'glotaran.builtin.models.doas.oscillation', 'glotaran.builtin.models.kinetic_image', 'glotaran.builtin.models.kinetic_image.initial_concentration', 'glotaran.builtin.models.kinetic_image.irf', 'glotaran.builtin.models.kinetic_image.k_matrix', 'glotaran.builtin.models.kinetic_image.kinetic_image_dataset_descriptor', 'glotaran.builtin.models.kinetic_image.kinetic_image_matrix', 'glotaran.builtin.models.kinetic_image.kinetic_image_megacomplex', 'glotaran.builtin.models.kinetic_image.kinetic_image_model', 'glotaran.builtin.models.kinetic_image.kinetic_image_result', 'glotaran.builtin.models.kinetic_spectrum', 'glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_dataset_descriptor', 'glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_model', 'glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_result', 'glotaran.builtin.models.kinetic_spectrum.spectral_constraints', 'glotaran.builtin.models.kinetic_spectrum.spectral_irf', 'glotaran.builtin.models.kinetic_spectrum.spectral_matrix', 'glotaran.builtin.models.kinetic_spectrum.spectral_penalties', 'glotaran.builtin.models.kinetic_spectrum.spectral_relations', 'glotaran.builtin.models.kinetic_spectrum.spectral_shape', 'glotaran.cli', 'glotaran.cli.commands', 'glotaran.cli.commands.explore', 'glotaran.cli.commands.export', 'glotaran.cli.commands.optimize', 'glotaran.cli.commands.pluginlist', 'glotaran.cli.commands.print', 'glotaran.cli.commands.util', 'glotaran.cli.commands.validate', 'glotaran.cli.main', 'glotaran.examples', 'glotaran.examples.sequential', 'glotaran.io', 'glotaran.io.prepare_dataset', 'glotaran.io.reader', 'glotaran.model', 'glotaran.model.dataset_descriptor', 'glotaran.model.model', 'glotaran.model.model_attribute', 'glotaran.model.model_decorator', 'glotaran.model.model_property', 'glotaran.model.util', 'glotaran.parameter', 'glotaran.parameter.parameter', 'glotaran.parameter.parameter_group', 'glotaran.parse', 'glotaran.parse.parser', 'glotaran.parse.register'] %}
{% block module %}

{% endblock %}
9 changes: 7 additions & 2 deletions docs/source/user_documentation/api_documentation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,19 @@ The API Documentation for glotaran is automatically created from its docstrings.
glotaran.analysis.matrix_calculation
glotaran.analysis.nnls
glotaran.analysis.optimize
glotaran.analysis.parameter_server
glotaran.analysis.problem_bag
glotaran.analysis.residual_calculation
glotaran.analysis.result
glotaran.analysis.scheme
glotaran.analysis.simulation
glotaran.analysis.variable_projection
glotaran.builtin
glotaran.builtin.file_formats
glotaran.builtin.file_formats.ascii
glotaran.builtin.file_formats.ascii.wavelength_time_explicit_file
glotaran.builtin.file_formats.sdt
glotaran.builtin.file_formats.sdt.sdt_file
glotaran.builtin.file_formats.sdt.sdt_file_reader
glotaran.builtin.models
glotaran.builtin.models.doas
glotaran.builtin.models.doas.doas_matrix
Expand All @@ -46,12 +51,12 @@ The API Documentation for glotaran is automatically created from its docstrings.
glotaran.builtin.models.kinetic_image.kinetic_image_result
glotaran.builtin.models.kinetic_spectrum
glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_dataset_descriptor
glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_matrix
glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_model
glotaran.builtin.models.kinetic_spectrum.kinetic_spectrum_result
glotaran.builtin.models.kinetic_spectrum.spectral_constraints
glotaran.builtin.models.kinetic_spectrum.spectral_irf
glotaran.builtin.models.kinetic_spectrum.spectral_matrix
glotaran.builtin.models.kinetic_spectrum.spectral_penalties
glotaran.builtin.models.kinetic_spectrum.spectral_relations
glotaran.builtin.models.kinetic_spectrum.spectral_shape
glotaran.cli
Expand Down
23 changes: 15 additions & 8 deletions glotaran/analysis/matrix_calculation.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,12 @@ def calculate_index_independend_ungrouped_matrices(scheme, parameter):
)
clp_labels[label] = clp_label
matrices[label] = matrix
if callable(model.constrain_matrix_function):
clp_label, matrix = model.constrain_matrix_function(parameter, clp_label, matrix, None)

if callable(model.has_matrix_constraints_function):
if model.has_matrix_constraints_function():
clp_label, matrix = \
model.constrain_matrix_function(parameter, clp_label, matrix, None)

constraint_labels_and_matrices[label] = LabelAndMatrix(clp_label, matrix)
return clp_labels, matrices, constraint_labels_and_matrices

Expand Down Expand Up @@ -72,10 +76,12 @@ def create_index_dependend_ungrouped_matrix_jobs(scheme, bag, parameter):
)
clp_labels[label].append(clp)
matrices[label].append(matrix)
if callable(model.constrain_matrix_function):
clp, matrix = dask.delayed(model.constrain_matrix_function, nout=2)(
parameter, clp, matrix, index
)

if callable(model.has_matrix_constraints_function):
if model.has_matrix_constraints_function():
clp, matrix = dask.delayed(model.constrain_matrix_function, nout=2)(
parameter, clp, matrix, index
)
constraint_labels_and_matrices[label].append((clp, matrix))

return clp_labels, matrices, constraint_labels_and_matrices
Expand Down Expand Up @@ -107,8 +113,9 @@ def get_matrices(result):
def constrain_and_combine_matrices(result):
matrices, index = result
clp, matrix = _combine_matrices(matrices)
if callable(model.constrain_matrix_function):
clp, matrix = model.constrain_matrix_function(parameter, clp, matrix, index)
if callable(model.has_matrix_constraints_function):
if model.has_matrix_constraints_function():
clp, matrix = model.constrain_matrix_function(parameter, clp, matrix, index)
return LabelAndMatrix(clp, matrix)

matrix_jobs = bag.map(calculate_group)
Expand Down
28 changes: 15 additions & 13 deletions glotaran/analysis/optimize.py
Original file line number Diff line number Diff line change
@@ -1,37 +1,39 @@
import collections
import dask
import dask.distributed as dd
import numpy as np
import lmfit
import numpy as np

from glotaran.parameter import ParameterGroup

from . import problem_bag, residual_calculation
from .result import Result
from .nnls import residual_nnls
from .variable_projection import residual_variable_projection
from .matrix_calculation import (
calculate_index_independend_ungrouped_matrices,
calculate_index_independend_grouped_matrices,
create_index_dependend_ungrouped_matrix_jobs,
create_index_dependend_grouped_matrix_jobs
)
from .nnls import residual_nnls
from .variable_projection import residual_variable_projection
from .result import Result

ResultFuture = \
collections.namedtuple('ResultFuture', 'bag clp_label matrix full_clp_label clp residual')


def optimize(scheme, verbose=True, client=None):

if client is None:
try:
client = dd.get_client()
except Exception:
client = dd.Client(processes=False)
initial_parameter = scheme.parameter.as_parameter_dict()
scheme = client.scatter(scheme)
optimization_result_future = client.submit(optimize_task, initial_parameter, scheme, verbose)
return optimization_result_future.result()

if client is not None:
scheme = client.scatter(scheme)
optimization_result_future = \
client.submit(optimize_task, initial_parameter, scheme, verbose)
result = optimization_result_future.result()

else:
result = optimize_task(initial_parameter, scheme, verbose)

return result


def optimize_task(initial_parameter, scheme, verbose):
Expand Down
44 changes: 22 additions & 22 deletions glotaran/analysis/residual_calculation.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,12 @@ def create_index_independend_ungrouped_residual(
residuals[label].append(residual)
penalties.append(residual)

if callable(scheme.model.additional_penalty_function):
additional_penalty = dask.delayed(scheme.model.additional_penalty_function)(
parameter, reduced_clp_labels[label], reduced_clps[label], i
)
penalties.append(additional_penalty)
if callable(scheme.model.has_additional_penalty_function):
if scheme.model.has_additional_penalty_function():
additional_penalty = dask.delayed(scheme.model.additional_penalty_function)(
parameter, reduced_clp_labels[label], reduced_clps[label], i
)
penalties.append(additional_penalty)

penalty = dask.delayed(np.concatenate)(penalties)
return reduced_clp_labels, reduced_clps, residuals, penalty
Expand Down Expand Up @@ -63,13 +64,12 @@ def create_index_dependend_ungrouped_residual(
residuals[label].append(residual)
penalties.append(residual)

# call removed because of performance reasons (issue #230)
# if callable(scheme.model.additional_penalty_function):
# additional_penalty = dask.delayed(scheme.model.additional_penalty_function)(
# parameter, clp_label, clp, i
# )
# penalties.append(additional_penalty)
# TODO: re-implement removed functionality (issue: #237)
if callable(scheme.model.has_additional_penalty_function):
if scheme.model.has_additional_penalty_function():
additional_penalty = dask.delayed(scheme.model.additional_penalty_function)(
parameter, clp_label, clp, i
)
penalties.append(additional_penalty)

penalty = dask.delayed(np.concatenate)(penalties)
return reduced_clp_labels, reduced_clps, residuals, penalty
Expand All @@ -86,11 +86,11 @@ def penalty_function(matrix_label, problem, labels_and_matrices):
clp, residual = residual_function(labels_and_matrices[matrix_label].matrix, problem.data)

penalty = residual
if callable(scheme.model.additional_penalty_function):
additional_penalty = scheme.model.additional_penalty_function(
parameter, labels_and_matrices[matrix_label].clp_label, clp, problem.index
)
if additional_penalty:
if callable(scheme.model.has_additional_penalty_function):
if scheme.model.has_additional_penalty_function():
additional_penalty = scheme.model.additional_penalty_function(
parameter, labels_and_matrices[matrix_label].clp_label, clp, problem.index
)
penalty = np.concatenate([penalty, additional_penalty])
return clp, residual, penalty
penalty_bag = \
Expand All @@ -113,11 +113,11 @@ def penalty_function(problem, labels_and_matrices):
clp, residual = residual_function(labels_and_matrices.matrix, problem.data)

penalty = residual
if callable(scheme.model.additional_penalty_function):
additional_penalty = scheme.model.additional_penalty_function(
parameter, labels_and_matrices.clp_label, clp, problem.index
)
if additional_penalty:
if callable(scheme.model.has_additional_penalty_function):
if scheme.model.has_additional_penalty_function():
additional_penalty = scheme.model.additional_penalty_function(
parameter, labels_and_matrices.clp_label, clp, problem.index
)
penalty = np.concatenate([penalty, additional_penalty])
return clp, residual, penalty
penalty_bag = \
Expand Down

0 comments on commit 3ca455d

Please sign in to comment.