[WIP] MAINT: Update optimize.shgo #14013

Stefan-Endres · 2021-05-09T22:09:27Z

Reference issues/fixes

Fixed an issue where an extra iteration is run before stopping the routine (resolves BUG: optimize.shgo: not using options #13974).
The simplicial sampling now generates the correct number of specified points.
Random and Sobol sampling with constraint functions can now handle args (should also resolve scipy.optimize.shgo(): 'args' is incorrectly passed to constraint function when 'sobol' sampling is used #12114).
At least one round-off is now used by default, as implied in the documentation.
Correctly handle objective functions that return the gradient alongside the objective value (resolves optimize.shgo: handle objective functions that return the gradient alongside the objective value #13547)
Fixed numerous typos in the documentation (Documentation typos and expanded examples Stefan-Endres/shgo#29).
Fixed cases where shgo was not using all possible stopping criteria (resolves BUG: optimize.shgo: not using options #13974).

Improved features

Parallelization is now implemented with the workers argument.
Added symmetry arguments which can improve computational performance up to a maximum of O(dim!) under ideal conditions.
Class based usage (similar to ENH: optimize: Class based Optimizers #8414) to improve usability. Example calling SHGO.iterate() until a "satisfactory" solution is found when no other stopping criteria is available (commonly requested feature).
Performance is improved due to various bug fixes. See the preliminary results below.
By default shgo should now return the minimum value or its round-off (resolves scipy.optimize.shgo fails on one-dimensional objective functions with particular symmetry #10538)
Improved speed of SHGO class initiation (serial).

Code streamlining

Use a single caching library for all sampling methods. Reduces number of code lines in main _shgo.py file to improve maintainability and readability.
-- (The main reason this PR for _shgo_lib is unfortunately so large is because adopting a new cache turned out to be the only way to solve the issue with the simplicial sampling method not iterating correctly).
Removed extra sobol point generations.
Fixed anti-patterns:
-- if a is 'teststring':
-- type(thing)

Benchmark profiles

As a preliminary test I reran the GO benchmarks for the simplicial method and compared it to the previous results quickly by overlaying a transparent window performance profiles over the previous results. As can be seen below the new simplicial method (darker blue solid line overlay) is noticeably faster than the previous results (lighter blue solid line).

The increased performance largely due to fixing the issue that required the sampling method to refine the entire complex instead of a fixed number of points.

Still to do is to rerun all the results and new sampling methods together with the other GO methods.

TODO:

ResourceWarning causing unittest test_18_parallelization to fail.
Reimplement adding local minimizer results to the simplicial complex.
Add shgo to scipy benchmarks (separate future pull request, Add shgo to optimize benchmarks #9451).
Fix importing error in scipy/stats/_hypotests.py#L7.
Possibly force using a fixed seed in Sobol and Halton sequences to improve deterministic performance.
Rerun benchmarks.
Still need to address documentation updates and improvments especially those discussed in Usability of optimize.shgo #13813, but as this pull request is.
Fix MAINT: optimize.shgo: returns incorrect solution to Rosenbrock problem #14533.

Proposed reviewers

@tupui @andyfaff

tupui · 2021-05-10T05:13:55Z

Thank you for all these improvements! I will try to have a look on it fast for this to have a chance to be in the release.

tupui

This is a huge update, thanks again for putting this together.

I did a first pass and I would first advise to clean these in order to facilitate the review. Things like reverting American spellings would lower the diff size a lot.
For workers try to follow the same usage as in differential_evolution. There is also a helper function to handler the pool (_util.MapWrapper).
Also, there are lot of TODOs. I would try to clean these as well (put them in the issue with a checklist instead).

scipy/optimize/_shgo.py

tupui · 2021-05-10T09:41:21Z

scipy/optimize/_shgo.py

+         minimizer_kwargs=None, options=None, sampling_method='simplicial',
+         workers=None):


New arguments should be keywords only.

Suggested change

minimizer_kwargs=None, options=None, sampling_method='simplicial',

workers=None):

minimizer_kwargs=None, options=None, sampling_method='simplicial', *,

workers=None):

tupui · 2021-05-10T09:43:18Z

scipy/optimize/_shgo.py

+            local minimisation routine every iteration. If False then only the
+            final minimiser pool will be run. Defaults to False.


American spelling with z should be used as the whole documentation uses optimize. Here and in other places.

Suggested change

local minimisation routine every iteration. If False then only the

final minimiser pool will be run. Defaults to False.

local minimization routine every iteration. If False then only the

final minimizer pool will be run. Defaults to False.

tupui · 2021-05-10T09:45:32Z

scipy/optimize/_shgo.py

+    workers : int  optional
+        Uses `multiprocessing.Pool <multiprocessing>`) to sample and run the 
+        local serial minimizatons in parrallel.


Add default and information about -1 (copy from differential_evolution).

tupui · 2021-05-10T09:49:11Z

scipy/optimize/_shgo.py

-         minimizer_kwargs=None, options=None, sampling_method='simplicial'):
+def shgo(func, bounds, args=(), constraints=None, n=100, iters=1, callback=None,
+         minimizer_kwargs=None, options=None, sampling_method='simplicial',
+         workers=None):


Should default to 1 here as differential_evolution and other ones.

tupui · 2021-05-10T09:59:26Z

scipy/optimize/_shgo.py

-                              self.symmetry, self.bounds, self.g_cons,
-                              self.g_args)
+        if self.disp:
+            print('Constructing and refining simplicial complex graph '


Print or logging? Same bellow.

tupui · 2021-05-10T10:01:53Z

scipy/optimize/_shgo.py

                    logging.info('=' * 30)

                if self.HC.V[x] not in self.minimizer_pool:
                    self.minimizer_pool.append(self.HC.V[x])

                if self.disp:
-                    logging.info('Neighbors:')
+                    logging.info('Neighbours:')


These are not needed either (while in this case it does not really matter, it increases the diff which makes it harder to review). In general we use American spelling.

tupui · 2021-05-10T10:04:59Z

scipy/optimize/_shgo.py

        results['x'] = self.xl_maps[ind_sorted[0]]  # Save global minima
        results['fun'] = self.f_maps[ind_sorted[0]]  # Save global fun value

        self.xl_maps = np.ndarray.tolist(self.xl_maps)
        self.f_maps = np.ndarray.tolist(self.f_maps)
        return results
+
+# TODO: In scipy version delete this


Also for caching there is functools.lru_cache

tupui · 2021-05-10T10:07:24Z

scipy/optimize/tests/test__shgo.py

 import pytest
 from pytest import raises as assert_raises, warns
+from nose.tools import nottest


Don't use nose. If you want to skip a test, use @pytest.mark.skip(reason="This is not run because ...")

tupui · 2021-05-10T10:11:07Z

scipy/optimize/_shgo_lib/_vertex.py

+                self.process_fpool = self.proc_fpool_g
+        else:
+            self.workers = workers
+            self.pool = mp.Pool(processes=workers)  #TODO: Move this pool to


Maybe rely on _util.MapWrapper instead

Stefan-Endres · 2021-05-10T12:08:25Z

Thank you very much @tupui . Apologies about the doctrings I think I accidentally pulled the wrong direction in the last merge. I will update and clean this as soon as possible to get it up to review standard.

tupui · 2021-05-10T12:27:12Z

Thank you very much @tupui . Apologies about the doctrings I think I accidentally pulled the wrong direction in the last merge. I will update and clean this as soon as possible to get it up to review standard.

No problem at all 😃

tupui · 2021-05-14T11:46:11Z

BTW, we recently added bounds to Nelder-Mead, so the solver_args needs to be updated. In general, maybe we should have this list generally accessible some place else. Otherwise we will always forget to update it here.

tupui · 2021-07-20T19:26:47Z

Hi @Stefan-Endres just checking in 😃 Any update?

Stefan-Endres · 2021-07-21T20:28:34Z

Hi, apologies for letting the time slip on this, I will have the weekend free to work on an hopefully finish this.

tupui · 2021-07-22T06:32:55Z

Hi, apologies for letting the time slip on this, I will have the weekend free to work on an hopefully finish this.

No problem 😃 Thanks!

tupui

I did another superficial pass.

There are unchecked item in your list. Are you still working on these or is the PR ready for more review?

Did you check that the resource usage is constant? (CPU, RAM). With more workers in the loop and the rewrite, things can go bad fast.

tupui · 2021-07-27T09:09:59Z

scipy/optimize/_shgo.py

@@ -241,7 +260,7 @@ def shgo(func, bounds, args=(), constraints=None, n=None, iters=1,
    specified using the `bounds` argument.

    While most of the theoretical advantages of SHGO are only proven for when
-    ``f(x)`` is a Lipschitz smooth function, the algorithm is also proven to
+    ``f(x)`` is a Lipschitz smooth function. the algorithm is also proven to


Suggested change

``f(x)`` is a Lipschitz smooth function. the algorithm is also proven to

``f(x)`` is a Lipschitz smooth function, the algorithm is also proven to

tupui · 2021-07-27T09:40:50Z

scipy/optimize/_shgo.py

+                          sfield=self.func, sfield_args=self.args,
+                          symmetry=self.symmetry,
+                          constraints=self.constraints,
+                          # constraints=self.g_cons,


Suggested change

# constraints=self.g_cons,

tupui · 2021-07-27T09:41:03Z

scipy/optimize/_shgo.py

        self.fn = 0  # Number of feasible sampling points evaluations performed
        self.hgr = 0  # Homology group rank

        # Default settings if no sampling criteria.
+        if (self.n is None) and (self.iters is None):
+            self.n = 128
+            self.nc = 0  # self.n


Suggested change

self.nc = 0 # self.n

self.nc = 0

tupui · 2021-07-27T09:41:19Z

scipy/optimize/_shgo.py

        self.n = n  # Sampling points per iteration
-        self.nc = n  # Sampling points to sample in current iteration
+        self.nc = 0  # n  # Sampling points to sample in current iteration


Suggested change

self.nc = 0 # n # Sampling points to sample in current iteration

self.nc = 0 # Sampling points to sample in current iteration

tupui · 2021-07-27T09:53:35Z

scipy/optimize/_shgo.py

+            if (not isinstance(constraints, tuple)) and (not
+            isinstance(constraints, list)):


Suggested change

if (not isinstance(constraints, tuple)) and (not

isinstance(constraints, list)):

if (not isinstance(constraints, tuple)) and \

(not isinstance(constraints, list)):

tupui · 2021-07-28T07:11:29Z

scipy/optimize/_shgo.py

+        # Generate sampling points.
+        # Generate uniform sample points in [0, 1]^m \subset R^m
+        if self.n_sampled == 0:
+            self.C = self.sobol_points(n, dim)


Where is sobol_points defined?

tupui · 2021-07-28T07:12:43Z

scipy/optimize/_shgo_lib/_complex.py

+import itertools
+import json
+import decimal
+from functools import lru_cache  # For Python 3 only


Suggested change

from functools import lru_cache # For Python 3 only

from functools import lru_cache

tupui · 2021-07-28T07:13:32Z

scipy/optimize/_shgo_lib/_complex.py

+            HC.V: The cache of vertices and their connection
+            HC.H: Storage structure of all vertex groups
+
+        :param dim: int, Spatial dimensionality of the complex R^dim


The doc is in the wrong format here, should be numpydoc.

tupui · 2021-07-28T07:14:57Z

scipy/optimize/_shgo_lib/_complex.py

+
+            # Printing
+            if printout:
+                print("=" * 19)


Some print left

tupui · 2021-07-28T07:16:32Z

scipy/optimize/_shgo_lib/_vertex.py

+"""
+Cache objects
+"""
+class VertexCacheBase(object):


Not needed anymore

Suggested change

class VertexCacheBase(object):

class VertexCacheBase:

Stefan-Endres · 2021-07-28T10:58:03Z

Hi @tupui apologies I should've commented that I need a bit more time before another review pass. Aside from the TODOs there is also a few more things to implement from your earlier comments in this PR.

I compared CPU times on the GO benchmarks, but not RAM usage yet I didn't think about that. I suppose the approriate measure would be to benchmark memory usage on the for workers=1 vs workers >=2 on the scipy GO testsuite, I will run these and post the results here.

tupui · 2021-07-28T11:15:25Z

Hi @tupui apologies I should've commented that I need a bit more time before another review pass. Aside from the TODOs there is also a few more things to implement from your earlier comments in this PR.

No worries, let me know when you're ready or need help with something 😃

I compared CPU times on the GO benchmarks, but not RAM usage yet I didn't think about that. I suppose the approriate measure would be to benchmark memory usage on the for workers=1 vs workers >=2 on the scipy GO testsuite, I will run these and post the results here.

Thanks! Would be great so that we have full confidence with all the changes.

dschmitz89 · 2022-01-22T10:04:24Z

@Stefan-Endres : I was thinking of adding support for the Bounds class for shgo. Is it worthwhile at the moment before this PR is merged?

Stefan-Endres · 2022-01-22T10:48:01Z

@Stefan-Endres : I was thinking of adding support for the Bounds class for shgo. Is it worthwhile at the moment before this PR is merged?

I think it's worth doing now as I would need to rebase this PR anyway, I am not certain when I will have full free weekend again to finish this.

tupui · 2022-01-24T11:33:26Z

FYI we've merged the improvement @Stefan-Endres.

wilrop · 2022-03-10T16:46:07Z

Hi, I'm just checking in to see if this update is already in the latest version of scipy because I'm still having problems of this kind. If it is already included, I can open a new issue so that maybe someone can look at it.

tupui · 2022-03-10T16:48:28Z

Hi, I'm just checking in to see if this update is already in the latest version of scipy because I'm still having problems of this kind. If it is already included, I can open a new issue so that maybe someone can look at it.

Hi @wilrop, thanks for checking in. Please don't open any new issue on this. This PR is still not merged and would address the issues. @Stefan-Endres is still working on it.

tupui · 2022-03-14T15:10:33Z

@Stefan-Endres I am adding a release milestone. We have until around June to finish this.

tupui · 2022-05-19T09:22:25Z

@Stefan-Endres FYI the branching for 1.9 is next week (26 of May). Let me know if I can help. Otherwise this will be pushed to 1.10.

Stefan-Endres · 2022-05-20T04:14:41Z

@tupui I will work on finishing this tonight. I think the only thing we really need is to test for memory leaks? The remaining issue #14533 could be done in a second PR.

tupui · 2022-05-20T08:11:53Z

Great thank you! I will be on the lookout to review. The memory leak is just (I hope) a sanity check, so yes just running it twice with a low count and with a high count of worker is fine.

Stefan-Endres · 2022-05-21T11:29:13Z

I ran into compilation rebasing this PR last night, currently running the test suite.

Some preliminary results indicate memory usage of multiprocessing is only about 0.025% bigger (expected due to bigger class initiated to handle the cache processing).

Running mem_tests.py:

import sys

from memory_profiler import profile
from pympler import asizeof
from scipy.optimize._shgo import SHGO
from scipy.optimize import rosen

# Define SHGO class
bounds = [(0,2), (0, 2), (0, 2), (0, 2)]
shc1 = SHGO(rosen, bounds, iters=4)  # With only 1 worker
shc2 = SHGO(rosen, bounds, iters=4, workers=2)  # With 2 workers using multiprocessing
shc1.iterate_all(), shc2.iterate_all()
print('Results with 1 cpu:')
print(f'asizeof.asizeof(shc1) (SHGO class) = {asizeof.asizeof(shc1)}')
print(f'asizeof.asizeof(shc1.HC) (Cache container) = {asizeof.asizeof(shc2.HC)}')
print('-')
print('Results with 2 cpus:')
print(f'asizeof.asizeof(shc2) (SHGO class) = {asizeof.asizeof(shc2)}')
print(f'asizeof.asizeof(shc2.HC) (Cache container) = {asizeof.asizeof(shc2.HC)}')
print('-')
print('Difference in memory usage:')
print(f'SHGO class: asizeof.asizeof(shc1) - asizeof.asizeof(shc2)  = {asizeof.asizeof(shc1) - asizeof.asizeof(shc2)}')
print(f'Cache containers: asizeof.asizeof(shc1.HC) - asizeof.asizeof(shc2.HC) = {asizeof.asizeof(shc1.HC) - asizeof.asizeof(shc2.HC)}')

produces

$ python mem_tests.py
Results with 1 cpu:
asizeof.asizeof(shc1) (SHGO class) = 29499688
asizeof.asizeof(shc1.HC) (Cache container) = 68495504
-
Results with 2 cpus:
asizeof.asizeof(shc2) (SHGO class) = 29517392
asizeof.asizeof(shc2.HC) (Cache container) = 68513208
-
Difference in memory usage:
SHGO class: asizeof.asizeof(shc1) - asizeof.asizeof(shc2)  = -17704
Cache containers: asizeof.asizeof(shc1.HC) - asizeof.asizeof(shc2.HC) = -17704

However, using the shgo library it is only about 0.003% bigger:

from shgo._shgo import SHGO  # Use shgo instead of scipy version
from scipy.optimize import rosen

Produces

$ python -i mem_tests.py 
Results with 1 cpu:
asizeof.asizeof(shc1) (SHGO class) = 29514720
asizeof.asizeof(shc1.HC) (Cache container) = 68510536
-
Results with 2 cpus:
asizeof.asizeof(shc2) (SHGO class) = 29516480
asizeof.asizeof(shc2.HC) (Cache container) = 68512264
-
Difference in memory usage:
SHGO class: asizeof.asizeof(shc1) - asizeof.asizeof(shc2)  = -1760
Cache containers: asizeof.asizeof(shc1.HC) - asizeof.asizeof(shc2.HC) = -1728

I don't fully understand why the scipy code which is idenitical has a bigger overhead for multiprocessing.

Next step is to finish rebasing and rerun the same tests on all the scipy GO benchmarks, please let me know if anyone prefers using different methods of measurement for testing the memory usage.

[ci skip]

…ns that occur for large parameter values (#14036) Avoid the numerical issues in `roots_jacobi`, `roots_gegenbauer`, and `_gen_roots_and_weights` that occur for large values of a and b in `roots_jacobi(n,a,b)`. Also helps with large values of n (i.e. n > 200). [ci skip]

Stefan-Endres · 2022-05-22T20:50:53Z

Apologies no idea why this happend, @tupui perhaps it would be better to close this PR and start a new one? Despite rebasing I still get this error when trying to build and run the tests:

RuntimeError: Running scipy/stats/_generate_pyx.py failed
Cythonizing sources
Traceback (most recent call last):
  File "/home/stefan_endres/projects/scipy_fucked/setup.py", line 534, in <module>
    setup_package()
  File "/home/stefan_endres/projects/scipy_fucked/setup.py", line 518, in setup_package
    generate_cython()
  File "/home/stefan_endres/projects/scipy_fucked/setup.py", line 267, in generate_cython
    raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!

tupui · 2022-05-23T03:20:36Z

Seems like you rebased agains a maintenance branch and not main.

@Stefan-Endres feel free to open a new PR if that's easier for you. FYI, since you opened this PR the infrastructure evolved a bit and we have a new convenient developer interface. If you have issues, I would suggest you looking into creating a new conda environment with the meson yaml and using python do.py.

Stefan-Endres · 2022-05-23T06:03:27Z

@Stefan-Endres feel free to open a new PR if that's easier for you.

I'm not sure how to undo the rebase and GitHub automatically adding a bunch of reviewer requests etc. I think it will be easier if you don't mind.

FYI, since you opened this PR the infrastructure evolved a bit and we have a new convenient developer interface. If you have issues, I would suggest you looking into creating a new conda environment with the meson yaml and using python do.py.

Thank you, I will try this after work tonight instead of the old build instructions.

tupui · 2022-05-23T07:45:40Z

Let me know if you need help. We can make a call as well 😃

wilrop · 2022-05-23T11:34:21Z

Hi, I'm probably super late to the party but I think there is a (different?) bug in how constraints are interpreted. If I add a single dictionary as a constraint everything is fine, but if I give a list of dictionaries as constraints it completely ignores all of them. Is this something that is handled with the current improvements or completely new? I've developed a minimal example, so I can create a new issue if that helps.

tupui · 2022-05-23T12:56:11Z

@wilrop if this bug is different from existing ones you are free to create an issue. Another possibility is that you wait a bit for an update here so we can try to reproduce your bug with latest changes (better IMO).

wilrop · 2022-05-23T12:58:26Z

Okay no problem, I'll wait and open an issue if the problem persists.

Stefan-Endres · 2022-05-23T18:10:12Z

@wilrop please open an issue with your code, there are unittests that have tuples of active constraints so this is possibly a new bug.

wilrop · 2022-05-24T06:55:13Z

I've just discovered that the bug is not specific to the list of constraints but rather with indexing in the x variable for constraints. I'll open a new issue anyway and tag @Stefan-Endres and @tupui as you have already been involved here.

Solution suggested by jjramsey in scipy#14589 (comment) The answer at the time was that an important refactor was in progress scipy#14013 This pull request is now closed and the bug persists in version 1.8.1 accordingly to this stack overflow question https://stackoverflow.com/q/72794609/12750353

Stefan-Endres requested a review from andyfaff as a code owner May 9, 2021 22:09

tupui added defect A clear bug or issue that prevents SciPy from being installed or used as expected scipy.optimize maintenance Items related to regular maintenance tasks enhancement A new feature or improvement labels May 10, 2021

tupui reviewed May 10, 2021

View reviewed changes

tupui reviewed Jul 28, 2021

View reviewed changes

Stefan-Endres mentioned this pull request Aug 3, 2021

MAINT: optimize.shgo: returns incorrect solution to Rosenbrock problem #14533

Closed

tupui added this to the 1.9.0 milestone Mar 14, 2022

tupui mentioned this pull request Apr 12, 2022

shgo error since scipy 1.8.0.dev0+1529.803e52d #14589

Closed

tupui mentioned this pull request May 19, 2022

BUG: scipy.optimize.shgo does not work with jac=True #16215

Closed

tylerjereddy and others added 6 commits May 22, 2022 22:44

MAINT: forward port 1.8.1 relnotes

1ec86d5

BLD: add a build option to force use of the g77 ABI with Meson (#16194)

a056e63

[ci skip]

TST: optimize: skip failing test of deprecated solver

5da2445

MAINT: Continue rebase

f9b0315

Add SLSQP to solver tests

c2bc9be

Stefan-Endres requested review from larsoner, rgommers, ev-br, person142, tylerjereddy, peterbell10, perimosocordiae and ilayn as code owners May 22, 2022 20:45

Stefan-Endres closed this May 22, 2022

mdhaber reopened this May 22, 2022

mdhaber changed the base branch from main to maintenance/0.5.2.x May 22, 2022 22:14

mdhaber changed the base branch from maintenance/0.5.2.x to main May 22, 2022 22:14

mdhaber closed this May 22, 2022

Stefan-Endres mentioned this pull request May 29, 2022

MAINT: Update optimize.shgo #16313

Merged

5 tasks

		minimizer_kwargs=None, options=None, sampling_method='simplicial',
		workers=None):

		local minimisation routine every iteration. If False then only the
		final minimiser pool will be run. Defaults to False.

	``f(x)`` is a Lipschitz smooth function. the algorithm is also proven to
	``f(x)`` is a Lipschitz smooth function, the algorithm is also proven to

	self.nc = 0 # n # Sampling points to sample in current iteration
	self.nc = 0 # Sampling points to sample in current iteration

		if (not isinstance(constraints, tuple)) and (not
		isinstance(constraints, list)):

	from functools import lru_cache # For Python 3 only
	from functools import lru_cache

[WIP] MAINT: Update optimize.shgo #14013

[WIP] MAINT: Update optimize.shgo #14013

Conversation

Stefan-Endres commented May 9, 2021 • edited

Reference issues/fixes

Improved features

Code streamlining

Benchmark profiles

TODO:

Proposed reviewers

tupui commented May 10, 2021

tupui left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stefan-Endres commented May 10, 2021

tupui commented May 10, 2021

tupui commented May 14, 2021

tupui commented Jul 20, 2021

Stefan-Endres commented Jul 21, 2021

tupui commented Jul 22, 2021

tupui left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stefan-Endres commented Jul 28, 2021

tupui commented Jul 28, 2021

dschmitz89 commented Jan 22, 2022

Stefan-Endres commented Jan 22, 2022

tupui commented Jan 24, 2022

wilrop commented Mar 10, 2022

tupui commented Mar 10, 2022

tupui commented Mar 14, 2022

tupui commented May 19, 2022

Stefan-Endres commented May 20, 2022 • edited

tupui commented May 20, 2022

Stefan-Endres commented May 21, 2022

Stefan-Endres commented May 22, 2022 • edited

tupui commented May 23, 2022 • edited

Stefan-Endres commented May 23, 2022

tupui commented May 23, 2022

wilrop commented May 23, 2022

tupui commented May 23, 2022

wilrop commented May 23, 2022

Stefan-Endres commented May 23, 2022

wilrop commented May 24, 2022 • edited

Stefan-Endres commented May 9, 2021 •

edited

Stefan-Endres commented May 20, 2022 •

edited

Stefan-Endres commented May 22, 2022 •

edited

tupui commented May 23, 2022 •

edited

wilrop commented May 24, 2022 •

edited