`**kwargs` input and `anesthetic.NestedSamples` output #90

AdamOrmondroyd · 2022-07-27T22:27:15Z

I've added a new interface pypolychord.run() which uses **kwargs rather than the PolyChordSettings object. It also returns an anesthetic.NestedSamples object rather than PolyChordOutput if anesthetic is already installed, otherwise it returns None.

The examples have been updated to use this interface.

Edit: run_pypolychord -> run_polychord. I'm glad the new interface uses just run!

…given to resume file

williamjameshandley

Many thanks for this @Ormorod, a more pythonic interface will likely definitely be useful, and the new kwargs treatment looks excellent (similar in spirit to #47)

However, as discussed via email we cannot change the run_pypolychord interface without breaking a lot of other people's code. You should write a new additional interface, e.g. run, which could be imported as from pypolychord import run, and leave run_pypolychord as-is.

You should also revert all the Python_Functions changes which can be kept as the legacy interface, particularly whilst we are stabilising the new interface. For example, there's no reason why nDerived, prior, and dumper can't be moved to *args or **kwargs

AdamOrmondroyd · 2022-07-28T09:01:11Z

Many thanks for this @Ormorod, a more pythonic interface will likely definitely be useful, and the new kwargs treatment looks excellent (similar in spirit to #47)

However, as discussed via email we cannot change the run_pypolychord interface without breaking a lot of other people's code. You should write a new additional interface, e.g. run, which could be imported as from pypolychord import run, and leave run_pypolychord as-is.

Ah yes I'd intended to put the old run_polychord() back in, temporarily removing it made the diff much more helpful.
btw the function doesn't have "py" in it, I've made this mistake many times.

You should also revert all the Python_Functions changes which can be kept as the legacy interface, particularly whilst we are stabilising the new interface. For example, there's no reason why nDerived, prior, and dumper can't be moved to *args or **kwargs

Okay, I just changed them all so I could test my changes. Would you like to keep the changes to run_pypolychord.py/.ipynb?

…s has any excess keys

williamjameshandley

Excellent work. The kwargs interface will be much more robust and future-proof.

In the new interface we should assume an anesthetic install, as it is now mature enough, and we should refine the example so that it is as pythonic and streamlined as possible, whilst showcasing what we can do (e.g. putting an anesthetic gui at the end).

Could we keep a copy of the old run_polychord.py, renamed to run_polychord_legacy_interface.py?

williamjameshandley · 2022-07-29T02:24:21Z

run_pypolychord.py

+kwargs = {
+    "nDerived": nDerived,
+    "prior": prior,
+    "dumper": dumper,
+    "file_root": 'gaussian',
+    "nlive": 200,
+    "do_clustering": True,
+    "read_resume": False,
+    "paramnames": paramnames,
+}


Could we pass these as keywords to run explicitly, rather than a dictionary?

run_pypolychord.py

pypolychord/polychord.py

run_pypolychord.py

run_pypolychord.ipynb

AdamOrmondroyd · 2024-01-09T22:57:37Z

@williamjameshandley I've taken the opportunity to add a few more test reflecting some of the recent PRs using the new interface, lmk if you think this should be its own PR

pypolychord/polychord.py

run_pypolychord.py

run_pypolychord.ipynb

AdamOrmondroyd · 2024-01-09T22:59:22Z

.github/workflows/CI.yml

@@ -75,7 +72,7 @@ jobs:
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
-        python -m pip install mpi4py pytest pytest-mpi fortranformat
+        python -m pip install mpi4py pytest pytest-mpi fortranformat anesthetic


We might not actually need pytest-mpi (which allows us to turn tests on or off depending on whether the test is mpirun) - do you see us ever wanting to test something on serial mode only?

we'll always want to test mpi, so leave this in

We can test with mpi without pytest-mpi, we only need it if we want to have tests which run specifically only with or without mpi

williamjameshandley

Looking very nice, many thanks @AdamOrmondroyd (and @appetrosyan for comments on interfacing over the years)

a few adjustments to consider/discuss inline

pypolychord/polychord.py

williamjameshandley · 2024-01-10T07:59:06Z

pypolychord/polychord.py

+        'nlives': {},
+        'seed': -1,
+    }
+    default_kwargs['grade_frac'] = [1.0]*len(default_kwargs['grade_dims'])


What does this line add, which 'grade_frac': [1.0] wouldn't?

grade_frac needs to have the same length as grade_dims (copied from the original interface, and forgetting to do this just failed the tests - glad we have them now!)

Just realised that this isn't quite right, since the default grade_frac should the same length as the input grade_dims, not the default grade_dims

fixed in cdaf063

williamjameshandley · 2024-01-10T07:59:52Z

pypolychord/polychord.py

+    if not set(kwargs.keys()) <= set(default_kwargs.keys()):
+        raise TypeError(f"{__name__} got unknown keyword arguments {kwargs.keys() - default_kwargs.keys()}")
+    default_kwargs.update(kwargs)
+    kwargs = default_kwargs


this is neat!

williamjameshandley · 2024-01-10T08:00:33Z

pypolychord/polychord.py

+    try:
+        if rank == 0:
+            os.makedirs(kwargs['base_dir'])
+    except OSError:
+        pass
+
+    try:
+        if rank == 0:
+            os.makedirs(os.path.join(kwargs['base_dir'], 'clusters'))
+    except OSError:
+        pass


Can these be combined?

I think no, as I believe the OSError expected here is that the directory already exists, and it may be the case that base_dir already exists, but base_dir/clusters does not.

I can do something neater with pathlib.Path, which can create several layers of directories at once, and can be told not to worry if the directory already exists

Done, I've also converted the other uses of os to pathlib

quickstart.py

williamjameshandley · 2024-01-10T08:11:32Z

.github/workflows/CI.yml

@@ -75,7 +72,7 @@ jobs:
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
-        python -m pip install mpi4py pytest pytest-mpi fortranformat
+        python -m pip install mpi4py pytest pytest-mpi fortranformat anesthetic


we'll always want to test mpi, so leave this in

README.rst

pypolychord/polychord.py

AdamOrmondroyd · 2024-01-10T13:38:47Z

pypolychord/polychord.py

+
+def _legacy_make_resume_file(settings, loglikelihood, prior):
+    kwargs = settings.__dict__
+    _make_resume_file(loglikelihood, prior = prior, **kwargs)


I realised I could dispense with the duplication between run and the old run_polychord with the same approach as here. @williamjameshandley what do you think?

@williamjameshandley could do with looking at this before I completely forget what I was thinking

…olyChordSettings (with difficulty)

AdamOrmondroyd · 2024-01-10T15:18:56Z

.github/workflows/CI.yml

@@ -75,7 +72,7 @@ jobs:
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
-        python -m pip install mpi4py pytest pytest-mpi fortranformat


maybe these should be added to requirements.txt (or requirements_test.txt)?

williamjameshandley · 2024-01-26T11:37:18Z

pypolychord/polychord.py

+    if rank == 0:
+        Path(settings.cluster_dir).mkdir(parents=True, exist_ok=True)


williamjameshandley

Looks good -- not sure I like import anesthetic as ac, but if you're happy please squash and merge.

williamjameshandley · 2024-01-26T11:58:56Z

quickstart.py

+    import anesthetic as ac
+    fig, ax = ac.make_2d_axes(['p0', 'p1', 'p2', 'p3', 'r'])


Can we change this to from anesthetic import make_2d_axes

I'm not a fan of cluttering the namespace personally, and I thought that Lukas and you liked ac as a shorthand for anesthetic!

I'll settle for import anesthetic then anesthetic.make_2d_axes

I've also imported numpy rather than clutter up the namespace with log and pi. Personally I've always found this annoying when I want to just substitute in a simple likelihood to check I haven't done anything daft, and copy-pasting the one from run_pypolychord.py is my go-to, but then I have to change log->np.log etc

I’ll weigh in on the discussion.

This is about consistency. If most of the project uses a “cluttered” namespace, it would be very distracting for users and other engineers.

The “cluttering” of a namespace makes it easy to substitute different versions of similar API elements: constants, variables, functions etc.

Consider if you needed to replace numpy.pi with scipy.pi. We’re talking a single line change in the imports compared to every single instance having to be renamed.

Another huge benefit IMO is readability. The namespace is not something the programmer sees. By contrast every place numpy functions or constants are used is by definition visible to the programmer. If there is no ambiguity, direct imports communicate more and make the program shorter. While sys.os functions may be harder to parse if imported directly, numpy and scipy functions are by definition known to the user.

I can see the convenience of substituting similar api elements, however iirc with anesthetic we were once caught out by the differences between the numpy and scipy Cholesky decompositions when we substituted one for the other

I understand what you mean about readability, but then it's also sometimes nice to be reminded where a function came from without going back to the preamble

However I will choose my battles and for the sake of finally seeing the back of this PR, I've done as Will originally suggested

However I will choose my battles and for the sake of finally seeing the back of this PR, I've done as Will originally suggested

Good choice.

That's usually the way it goes when you're contributing. The maintainers set style and convention, and in the few situations where I've seen maintainers change their mind about either of those things, it was a separate discussion that came with its own PR.

I understand what you mean about readability, but then it's also sometimes nice to be reminded where a function came from without going back to the preamble

Very much up to the maintainer. It's a question of style, and each style has trade-offs. On that note, If I were curious about where a function came from, most all code editors (Emacs, Vim etc.) have a facility that will display the documentation for a symbol if you hover. There's also code lenses that will optionally qualify all functions regardless of the actual text of the program.

I can see the convenience of substituting similar api elements, however iirc with anesthetic we were once caught out by the differences between the numpy and scipy Cholesky decompositions when we substituted one for the other

Sure. But that isn't exactly an argument for qualifying the paths.

First, as a result of that debacle, you know that the two are not the same, and that you do need to disambiguate, this is valuable information if you plan to contribute/co-maintain a project.

Secondly, there's more than one way to disambiguate; only one way to do it is to qualify the paths. Another way to do it, and I'll argue better, is to disambiguate by intent: np.cholesky → upper_triangular, and scipy → lower_triangular. Now your code communicates information to the user, without them having to dig through documentation to find out that the difference is in the defaults.

… and pi

…with log and pi" This reverts commit cc807f5.

AdamOrmondroyd and others added 15 commits May 19, 2022 17:15

neighbor spelling (assuming we're sticking to American spelling!)

f1f52ed

Merge branch 'PolyChord:master' into master

2f0c339

updated run_poypolychord to use **kwargs rather than PolyChordSettings

6efc40c

return anesthetic.NestedSamples output if installed

df7b300

added kwargs to run_pypolychord() docstring

3f21a7b

changed examples to use **kwargs

85b7dfb

accidental rename of run_polychord()

8cfcd22

set kwarg defaults separately to run_pypolychord so that they can be …

84c30c3

…given to resume file

run_polychord creates the paramnames file

6fe0516

changed other examples paramnames are passed to run_pypolychord

51928d0

mention anesthetic output in docstring (incomplete)

da44aa7

tidied up grade_dims and nlives argument processing

617ad12

logx should be logc

d375810

missing comma

6270512

removed settings argument

1316fa8

williamjameshandley requested changes Jul 28, 2022

View reviewed changes

AdamOrmondroyd added 12 commits July 28, 2022 10:03

Restored old interface

28ee39b

run_polychord -> run

50fa72b

reverted Python_Functions/ (apart from logx->logc correction

ae5c261

put PolyChordSettings back in __init__.py

247c51c

run() uses make_resume_file_kwargs_interface

cf54330

added run to __init__.py

dd2cf63

update nDerived, prior and dumper to kwargs

7c06066

nDims is not a kwarg

7b79d66

add nDerived, prior and dumper to kwargs

9698948

use default_kwargs to set the defaults, then use this to see if kwarg…

5117092

…s has any excess keys

separate default_kwargs that refer to other default_kwargs

aa64442

wrong brackets - think it's time to leave this for a moment

0f153cb

williamjameshandley requested changes Jul 29, 2022

View reviewed changes

AdamOrmondroyd commented Jan 9, 2024

View reviewed changes

some test housekeeping

ae0bb6d

williamjameshandley requested changes Jan 10, 2024

View reviewed changes

AdamOrmondroyd added 2 commits January 10, 2024 13:22

Merge remote-tracking branch 'upstream/master' into anesthetic_output

be5e7c9

use or between the two loglikelihood return options

8143c0a

AdamOrmondroyd commented Jan 10, 2024

View reviewed changes

AdamOrmondroyd added 12 commits January 10, 2024 13:51

use pathlib.Path to save OSError nonsense

3465c20

explain paramnames more clearly

9885121

check that grade_dims add up to nDims and test

82add9d

update run docstring anesthetic stuff

b9cc4fe

correct run_pypolychord return

2cae544

correct missing blank lines in run() docstring

9b4be33

add cluster_dir to kwargs, since technically it could be changed in P…

45e9f8c

…olyChordSettings (with difficulty)

remove os entirely in favour of pathlib

387b0d0

fix line too long

4f5270f

update date

a17d78d

fix default grade_frac

cdaf063

remove pytest-mpi requirement from CI

40cb559

AdamOrmondroyd commented Jan 10, 2024

View reviewed changes

AdamOrmondroyd changed the title ~~anesthetic.NestedSamples output and **kwargs input~~ **kwargs input and anesthetic.NestedSamples output Jan 25, 2024

remove unecessary cast to set

d420526

williamjameshandley reviewed Jan 26, 2024

View reviewed changes

williamjameshandley previously approved these changes Jan 26, 2024

View reviewed changes

don't use ac shorthand, but also don't clutter the namespace with log…

cc807f5

… and pi

AdamOrmondroyd dismissed williamjameshandley’s stale review via cc807f5 January 27, 2024 12:35

AdamOrmondroyd added 2 commits January 28, 2024 22:58

Revert "don't use ac shorthand, but also don't clutter the namespace …

6d735e5

…with log and pi" This reverts commit cc807f5.

from anesthetic import make_2d_axes

5d9dadf

williamjameshandley approved these changes Jan 29, 2024

View reviewed changes

AdamOrmondroyd merged commit b3daab9 into PolyChord:master Jan 29, 2024
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`**kwargs` input and `anesthetic.NestedSamples` output #90

`**kwargs` input and `anesthetic.NestedSamples` output #90

AdamOrmondroyd commented Jul 27, 2022 •

edited

Loading

williamjameshandley left a comment

AdamOrmondroyd commented Jul 28, 2022

williamjameshandley left a comment

williamjameshandley Jul 29, 2022

AdamOrmondroyd commented Jan 9, 2024

AdamOrmondroyd Jan 9, 2024

williamjameshandley Jan 10, 2024

AdamOrmondroyd Jan 10, 2024

williamjameshandley left a comment

williamjameshandley Jan 10, 2024

AdamOrmondroyd Jan 10, 2024 •

edited

Loading

AdamOrmondroyd Jan 10, 2024

AdamOrmondroyd Jan 10, 2024

williamjameshandley Jan 10, 2024

williamjameshandley Jan 10, 2024

AdamOrmondroyd Jan 10, 2024

AdamOrmondroyd Jan 10, 2024

williamjameshandley Jan 10, 2024

AdamOrmondroyd Jan 10, 2024 •

edited

Loading

AdamOrmondroyd Jan 25, 2024

AdamOrmondroyd Jan 10, 2024

williamjameshandley Jan 26, 2024

williamjameshandley left a comment

williamjameshandley Jan 26, 2024

AdamOrmondroyd Jan 26, 2024

AdamOrmondroyd Jan 27, 2024

appetrosyan Jan 27, 2024

AdamOrmondroyd Jan 28, 2024 •

edited

Loading

appetrosyan Jan 29, 2024

		if rank == 0:
		Path(settings.cluster_dir).mkdir(parents=True, exist_ok=True)

		import anesthetic as ac
		fig, ax = ac.make_2d_axes(['p0', 'p1', 'p2', 'p3', 'r'])

**kwargs input and anesthetic.NestedSamples output #90

**kwargs input and anesthetic.NestedSamples output #90

Conversation

AdamOrmondroyd commented Jul 27, 2022 • edited Loading

williamjameshandley left a comment

Choose a reason for hiding this comment

AdamOrmondroyd commented Jul 28, 2022

williamjameshandley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdamOrmondroyd commented Jan 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

williamjameshandley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdamOrmondroyd Jan 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdamOrmondroyd Jan 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

williamjameshandley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdamOrmondroyd Jan 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

`**kwargs` input and `anesthetic.NestedSamples` output #90

`**kwargs` input and `anesthetic.NestedSamples` output #90

AdamOrmondroyd commented Jul 27, 2022 •

edited

Loading

AdamOrmondroyd Jan 10, 2024 •

edited

Loading

AdamOrmondroyd Jan 10, 2024 •

edited

Loading

AdamOrmondroyd Jan 28, 2024 •

edited

Loading