Finish 0.20.1

.github/workflows/ci.yml

@@ -10,7 +10,7 @@ on:
     tags:
       - "*"
     branches:
-      - master
+      - main
       - develop
       - "release/*"
       - "hotfix/*"
@@ -22,7 +22,7 @@ on:
 jobs:
   unittests:
     runs-on: ${{ matrix.os }}
-    timeout-minutes: 240 # for eager mode
+    timeout-minutes: 200 # for eager mode
     name: tests on ${{ matrix.os }} with ${{ matrix.python-version }} compiled = ${{ matrix.use-graph }}
     env:
       ZFIT_DO_JIT: ${{ matrix.use-graph }}
@@ -88,7 +88,7 @@ jobs:
         run: |
           pytest --cov=. --cov-config=pyproject.toml -n2 --cov-report=xml --cov-report=term-missing
 
-#          pytest --cov=. --cov-config=pyproject.toml -n${{ steps.cpu-cores.outputs.count }} --cov-report=xml --cov-report=term-missing
+#          pytest --cov=. --cov-config=pyproject.toml -n auto --cov-report=xml --cov-report=term-missing  # logical number of cpus, falls back to auto
   #               TODO: fails on windows? add again above pytest?
   #          if [ "$GITHUB_REF" == 'refs/heads/develop' ]; then
   #            echo "PYTEST_ADDOPTS=--longtests-kde --longtests" >> "$GITHUB_ENV"

.pre-commit-config.yaml

@@ -3,6 +3,7 @@ ci:
 #  skip:
 #    - oxipng
 
+
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.5.0
@@ -21,10 +22,13 @@ repos:
       - id: requirements-txt-fixer
       - id: trailing-whitespace
         files: ".*\\.(?:tex|py)$"
+        args: [--markdown-linebreak-ext=md]
+        exclude: (^notebooks/|^tests/truth/)
       - id: detect-private-key
       - id: fix-byte-order-marker
       - id: check-ast
       - id: check-docstring-first
+      - id: debug-statements
   - repo: local
     hooks:
       - id: doc arg replacer
@@ -105,7 +109,7 @@ repos:
 
 
   - repo: https://github.com/python-jsonschema/check-jsonschema
-    rev: 0.28.1
+    rev: 0.28.2
     hooks:
       - id: check-github-workflows
       - id: check-github-actions
@@ -139,7 +143,7 @@ repos:
 
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: "v0.3.5"
+    rev: "v0.3.7"
     hooks:
       - id: ruff
         types_or: [ python, pyi, jupyter ]

CHANGELOG.rst

@@ -4,29 +4,18 @@ Changelog
 
 .. _newest-changelog:
 
-Develop
+0.20.1 (14 Apr 2024)
 ========================
 
 Major Features and Improvements
 -------------------------------
-
-Breaking changes
-------------------
-
-Deprecations
--------------
+- fix dumping and add convenience wrapper ``zfit.dill`` to dump and load objects with dill (a more powerful pickle). This way, any zfit object can be saved and loaded, such as ``FitResult`` that contains all other important objects to recreate the fit.
+- improved performance for numerical gradient calculation, fixing also a minor numerical issue.
 
 Bug fixes and small changes
 ---------------------------
+- runing binned fits without a graph could deadlock, fixed.
 
-Experimental
-------------
-
-Requirement changes
--------------------
-
-Thanks
-------
 
 0.20.0 (12 Apr 2024)
 ========================
@@ -54,7 +43,7 @@ Major Features and Improvements
 - add ``StudentT`` PDF, the standard Student's t distribution, taken from `tensorflow-probability implementation <https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/StudentT>`_.
 - add ``GaussExpTail`` and ``GeneralizedGaussExpTail`` PDFs, which are a Gaussian with an exponential tail on one side and a Gaussian with different sigmas on each side and different exponential tails on each side respectively.
 - add ``QGauss`` PDF, a distribution that arises from the maximization of the Tsallis entropy under appropriate constraints, see `here <https://en.wikipedia.org/wiki/Q-Gaussian_distribution>`_.
-- add ``BifurGauss` PDF, a Gaussian distribution with different sigmas on each side of the mean.
+- add ``BifurGauss`` PDF, a Gaussian distribution with different sigmas on each side of the mean.
 - add ``Bernstein`` PDF, which is a PDF defined by a linear combination of Bernstein polynomials given their coefficients.
 - add ``Gamma`` PDF, the Gamma distribution.
 - ``Data`` has now a ``with_weights`` method that returns a new data object with different weights and an improved ``with_obs`` that allows to set obs with new limits. These replace the ``set_weights`` and ``set_data_range`` methods for a more functional approach.
@@ -77,7 +66,7 @@ This release contains multiple "breaking changes", however, the vast majority if
 - ``Data.from_root``: deprecated arguments ``branches`` and ``branch_aliases`` have been removed. Use ``obs`` and ``obs_aliases`` instead.
 - ``NameAlreadyTakenError`` was removed, see above for the new behavior. This should not have an effect on any existing code *except if you relied on the error being thrown*.
 - Data objects had an intrinsic, TensorFlow V1 legacy behavior: they were actually cut when the data was *retrieved*. This is now changed and the data is cut when it is created. This should not have any impact on existing code and just improve runtime and memory usage.
-- Partial integration used to use some broadcasting tricks that could potentially fail. It uses now a dynamic while loop that _could_ be slower but works for arbitrary PDFs. This should not have any impact on existing code and just improve stability (but technically, the data given to the PDF _if doing partial integration_ is now "different", in the sense that it's now not different anymore from any other call)
+- Partial integration used to use some broadcasting tricks that could potentially fail. It uses now a dynamic while loop that _could_ be slower but works for arbitrary PDFs. This should not have any impact on existing code and just improve stability (but technically, the data given to the PDF *if doing partial integration* is now "different", in the sense that it's now not different anymore from any other call)
 - if a ``tf.Variable`` was used to store the number of sampled values in a sampler, it was possible to change the value of that variable to change the number of samples drawn. This is now not possible anymore and the number of samples should be given as an argument ``n`` to the ``resample`` method, as was possible since a long time.
 - ``create_sampler`` has a breaking change for ``fixed_params``: when the argument was set to False, any change in the parameters would be reflected when resampling.
   This highly statebased behavior was confusing and is now removed. The argument is now called ``params``
@@ -126,7 +115,7 @@ Requirement changes
 
 Thanks
 ------
-- huge thanks to @iasonkrommydas for the addition of various PDFs and to welcome him on board as a new contributor!
+- huge thanks to @ikrommyd (Iason Krommydas) for the addition of various PDFs and to welcome him on board as a new contributor!
 - @anjabeck for the addition of the ``ChiSquared`` PDF
 
 0.18.2 (13 Mar 2024)
@@ -271,7 +260,7 @@ Bug fixes and small changes
 Requirement changes
 -------------------
 - update to TensorFlow 2.12
-- removed tf_quant_finance
+- removed ``tf_quant_finance``
 
 
 0.13.2 (15. June 2023)
@@ -313,7 +302,7 @@ Bug fixes and small changes
 
 Experimental
 ------------
-- Added support on a best-effort for human-readable serialization of objects including an HS3-like representation, find a `tutorial on serialization here<https://zfit-tutorials.readthedocs.io/en/latest/tutorials/components/README.html#serialization>`_. Most built-in unbinned PDFs are supported. This is still experimental and not yet fully supported. Dumping can be performed safely, loading maybe easily breaks (also between versions), so do not rely on it yet. Everything else - apart of trying to dump - should only be used for playing around and giving feedback purposes.
+- Added support on a best-effort for human-readable serialization of objects including an HS3-like representation, find a `tutorial on serialization here <https://zfit-tutorials.readthedocs.io/en/latest/tutorials/components/README.html#serialization>`_. Most built-in unbinned PDFs are supported. This is still experimental and not yet fully supported. Dumping can be performed safely, loading maybe easily breaks (also between versions), so do not rely on it yet. Everything else - apart of trying to dump - should only be used for playing around and giving feedback purposes.
 
 Requirement changes
 -------------------
@@ -480,11 +469,11 @@ Major Features and Improvements
 Deprecations
 -------------
 - the default name of the uncertainty methods ``hesse`` and ``errors`` depended on
-  the method used (such as 'minuit_hesse', 'zfit_errors' etc.) and would be the exact method name.
+  the method used (such as ``"minuit_hesse"``, ``"zfit_errors"`` etc.) and would be the exact method name.
   New names are now 'hesse' and 'errors', independent of the method used. This reflects better that the
   methods, while internally different, produce the same result.
   To update, use 'hesse' instead of 'minuit_hesse' or 'hesse_np' and 'errors' instead of 'zfit_errors'
-  or 'minuit_minos' in order to access the uncertainties in the fitresult.
+  or ``"minuit_minos"`` in order to access the uncertainties in the fitresult.
   Currently, the old names are still available for backwards compatibility.
   If a name was explicitly chosen in the error method, nothing changed.
 
@@ -846,7 +835,7 @@ Requirement changes
 -------------------
 
 - TensorFlow==2.3 (before 2.2)
-- tensorflow_probability==0.11
+- ``tensorflow_probability==0.11``
 - tensorflow-addons  # spline interpolation in convolution
 
 
@@ -970,7 +959,7 @@ Major Features and Improvements
 
   To test if a value is inside a space, ``Space.inside`` can be used. To filter values, ``Space.filter``.
 
-  The limits returned are now by default numpy arrays with the shape (1, n_obs). This corresponds well
+  The limits returned are now by default numpy arrays with the shape ``(1, n_obs)``. This corresponds well
   to the old layout and can, using ``z.unstack_x(lower)`` be treated like ``Data``. This has also some
   consequences for the output format of ``rect_area``: this is now a vector.
 
@@ -1048,7 +1037,7 @@ Bug fixes and small changes
 
 Experimental
 ------------
-- added a new error method, 'zfit_error' that is equivalent to 'minuit_minos', but not fully
+- added a new error method, ``'zfit_error'`` that is equivalent to ``'minuit_minos'``, but not fully
   stable. It can be used with other minimizers as well, not only Minuit.
 
 Requirement changes
@@ -1279,7 +1268,7 @@ Fixed Partial numeric integration
 Bugfixes mostly, a few major fixes. Partial numeric integration works now.
 
 Bugfixes
- - data_range cuts are now applied correctly, also in several dimensions when a subset is selected
+ - ``data_range`` cuts are now applied correctly, also in several dimensions when a subset is selected
    (which happens internally of some Functors, e.g. ProductPDF). Before, only the selected obs was respected for cuts.
  - parital integration had a wrong take on checking limits (now uses supports).
 
@@ -1291,7 +1280,7 @@ With 0.3.2, bugfixes and three changes in the API/behavior
 
 Breaking changes
 ----------------
- - tfp distributions wrapping is now different with dist_kwargs allowing for non-Parameter arguments (like other dists)
+ - tfp distributions wrapping is now different with ``dist_kwargs`` allowing for non-Parameter arguments (like other dists)
  - sampling allows now for importance sampling (sampler in Model specified differently)
  - ``model.sample`` now also returns a tensor, being consistent with ``pdf`` and ``integrate``
 

README.rst

@@ -8,8 +8,8 @@ zfit: scalable pythonic fitting
    :target: https://scikit-hep.org
 
 .. image:: https://img.shields.io/badge/DOI-10.1016%2Fj.softx.2020.100508-yellow
-   :target: https://www.sciencedirect.com/science/article/pii/S2352711019303851  
-   :alt: DOI 10.1016/j.softx.2020.100508  
+   :target: https://www.sciencedirect.com/science/article/pii/S2352711019303851
+   :alt: DOI 10.1016/j.softx.2020.100508
 
 .. image:: https://img.shields.io/pypi/pyversions/zfit
    :target: https://pypi.org/project/zfit/
@@ -71,7 +71,7 @@ zfit is available on pip and conda-forge. To install it (recommended: use a virt
 .. code-block:: bash
 
    pip install -U zfit[all]
-   
+
 (the ``-U`` just indicates to upgrade zfit, in case you have it already installed)
 or for minimal dependencies
 
@@ -116,7 +116,7 @@ Example in short
 
     # minimize
     minimizer = zfit.minimize.Minuit()
-    result = minimizer.minimize(nll)
+    result = minimizer.minimize(nll).update_params()
 
     # calculate errors
     param_errors = result.hesse()
@@ -173,7 +173,7 @@ Fits are performed in three steps:
     minimizer = zfit.minimize.Minuit()
 
     # Stage 3: minimise the given negative log-likelihood
-    result = minimizer.minimize(nll)
+    result = minimizer.minimize(nll).update_params()
 
 Errors are calculated with a further function call to avoid running potentially expensive operations if not needed:
 

docs/conf.py

@@ -83,10 +83,15 @@
 atexit.register(lambda path=zfit_tutorials_path: shutil.rmtree(path))
 pygit2.clone_repository("https://github.com/zfit/zfit-tutorials", zfit_tutorials_path)
 
+# todo: do we need it and connect?
+# zfit_physics_path = project_dir / "docs" / "_tmp" / "zfit-physics"
+# atexit.register(lambda path=zfit_physics_path: shutil.rmtree(path))
+# pygit2.clone_repository("https://github.com/zfit/zfit-physics", zfit_physics_path)
+
 zfit_images_path = project_dir.joinpath("docs", "images")
 docs_images_path = project_dir.joinpath("docs", "_static", "images")
-docs_images_path.mkdir(parents=True, exist_ok=True)
 atexit.register(lambda path=docs_images_path: shutil.rmtree(path))
+docs_images_path.mkdir(parents=True, exist_ok=True)
 shutil.copytree(zfit_images_path, docs_images_path, dirs_exist_ok=True)
 
 nb_execution_in_temp = True

docs/images/README.rst

@@ -1,2 +1,2 @@
-The _generated folder contains images that are generated by running pytest with
+The ``_generated`` folder contains images that are generated by running pytest with
 ``pytest -m plots`` which will only run the tests that actually plot something.

docs/plots/fftconv_spline_linear.py

@@ -2,6 +2,7 @@
 from __future__ import annotations
 
 import matplotlib.pyplot as plt
+import pytest
 import tensorflow as tf
 
 import zfit
@@ -30,7 +31,7 @@ def plot_conv_comparison():
     plt.plot(x, probs_spline, label="spline")
     plt.legend()
     plt.title(f"FFT Conv with interpolation: {n_points_conv} points")
-    plt.show(block=True)
+    pytest.zfit_savefig(folder="fftconv_spline_linear")
 
 
 if __name__ == "__main__":

docs/tutorials/components/intro/data.rst

@@ -39,7 +39,7 @@ The ``branches_alias`` can be seen as a list of strings that renames the origina
 
     The implementation of the ``from_root`` method makes uses of the uproot packages,
     which uses Numpy to cast blocks of data from the ROOT file as Numpy arrays in time optimised manner.
-    This also means that the *goodies* from uproot can also be used by specifying the root_dir_options,
+    This also means that the *goodies* from uproot can also be used by specifying the ``root_dir_options``,
     such as cuts in the dataset. However, this can be applied later when examining the produced dataset
     and it is the advised implementation of this.
 

docs/tutorials/components/intro/model.rst

@@ -84,7 +84,7 @@ Another feature for the PDF is to calculate its integral in a certain limit. Thi
     print(f"Integral={integral_norm}")
 
 In this case the CB has been normalised using the range defined in the observable.
-Conversely, the ``norm_range`` in which the PDF is normalised can also be specified as input.
+Conversely, the ``norm`` in which the PDF is normalised can also be specified as input.
 
 Composite PDF
 -------------

docs/tutorials/components/intro/parameter.rst

@@ -36,7 +36,7 @@ clip the value to the limits given by :py:meth:`~zfit.Parameter.lower_limit` and
 
 The value of the parameter can be changed with the :py:func:`~zfit.Parameter.set_value` method.
 Using this method as a context manager, the value can also temporarily changed.
-However, be aware that anything _dependent_ on the parameter will have a value with the
+However, be aware that anything *dependent* on the parameter will have a value with the
 parameter evaluated with the new value at run-time:
 
 .. jupyter-execute::

docs/tutorials/components/intro/space.rst

@@ -44,7 +44,7 @@ with or without limits.
  * **rectangular**: This type is the usual limit as e.g. ``(-2, 5)`` for a simple, 1 dimensional interval.
 
 A space should usually be created with limits that define the default space of an object.
-This correspond for example to the default normalization range ``norm_range`` or sampling range.
+This correspond for example to the default normalization range ``norm`` or sampling range.
 
 .. code-block::
 

docs/tutorials/components/intro/toy_studies.rst

@@ -82,24 +82,3 @@ If some parameters are constrained to values observed from external measurements
 then sampling of the observed values might be needed to obtain an unbiased sample from the model. Example:
 
 TODO: the sample below is not correct and needs updating...
-
-.. jupyter-execute::
-
-    # same model depending on mu1, sigma1, mu2, sigma2
-
-    constraint = zfit.constraint.GaussianConstraint(params=[sigma1, sigma2],
-                                                    observation=[1.0, 0.5],
-                                                    uncertainty=[0.1, 0.05])
-
-    n_samples = 5
-
-    sampler = model.create_sampler(n=n_samples)
-    nll = zfit.loss.UnbinnedNLL(model=model, data=sampler, constraints=constraint)
-
-    constr_values = constraint.sample(n=n_samples)
-
-    for constr_params, constr_vals in constr_values.items():
-        sampler.resample()
-        # do something with nll, temporarily assigning values to the parameters
-        with zfit.param.set_values(constr_params, constr_vals):
-            minimizer.minimize(nll)  # minimize

docs/user_api/pdf/physics.rst

@@ -0,0 +1,12 @@
+
+
+.. autosummary::
+    :toctree: _generated/physics
+
+    zfit_physics.pdf.Argus
+    zfit_physics.pdf.RelativisticBreitWigner
+    zfit_physics.pdf.CMSShape
+    zfit_physics.pdf.Cruijff
+    zfit_physics.pdf.ErfExp
+    zfit_physics.pdf.Novosibirsk
+    zfit_physics.pdf.Tsallis

docs/user_api/zfit.dill.rst

@@ -0,0 +1,18 @@
+dill
+--------------------------------------------------------------------------------
+
+Drop-in replacement for `dill <https://github.com/uqfoundation/dill#dill>`_, a more powerful version of pickle: Contains a thin wrapper around dumping objects with dill.
+
+Simply use ``import zfit.dill as dill`` instead of ``import dill`` to use the zfit version of dill or use the functions directly with ``zfit.dill.dump`` and ``zfit.dill.dumps`` (and others).
+
+In general, ``dill.dump`` and ``dill.dumps`` work directly with zfit objects. However, due to some issue with the
+garbage collection, they can fail unpredictably. This module, mostly the functions ``zfit.dill.dump`` and ``zfit.dill.dumps`` are thin wrappers around dill that circumvent this issue if it arises.
+
+
+The module contains *all* functions and classes of dill. Here, only the altered functions are documented.
+
+.. autosummary::
+    :toctree: _generated/dill
+
+    zfit.dill.dump
+    zfit.dill.dumps

docs/user_api/zfit.dimension.rst

@@ -8,5 +8,3 @@ Dimension
 
     zfit.dimension.Space
     zfit.dimension.combine_spaces
-    zfit.dimension.add_spaces
-    zfit.dimension.Coordinates

docs/user_api/zfit.pdf.rst

@@ -78,3 +78,15 @@ They offer more or less support and freedom.
     :maxdepth: 2
 
     pdf/custom_base
+
+Physics PDFs
+##############
+
+Physics PDFs are PDFs that are often used in high energy physics. They are in the
+zfit-physics package, which needs to be installed separately.
+
+
+.. toctree::
+    :maxdepth: 2
+
+    pdf/physics