Create a better interface for switching precision and rounding

[skirpichev]

The mpmath provides global context objects (mp, fp, iv). Most functions in the global mpmath namespace are actually methods of the mp context.

It would be better, if we could introduce an explicit notion of the current context, that will prevent issues like #657, something more close to gmpy2 contexts handling. (Another example that worth considering is the figfloat package.)

The current interface also is not thread-safe.

[oscarbenjamin]

The thread-safe part can be addressed in a minimal way by ensuring that either the contexts or the attributes of the context use thread-local variables:
https://docs.python.org/3/library/threading.html#thread-local-data

That would entail some slowdown on attribute access but it is not clear how significant it is in the context of everything else. The slowdown of thread-local data access in Python is not as much as it would be when working in something like C which has less general overhead.

A thread-local is really just a global variable that is per-thread rather than being global to all threads. It is thread-safe but still otherwise has all of the pitfalls of global variables so it would be good to make it clear in the docs how to avoid using this altogether at least for downstream library code that really should not depend on using global state like this.

It is broadly possible to just use local contexts:

In [30]: from mpmath import MPContext

In [31]: mp1 = MPContext()

In [32]: mp2 = MPContext()

In [33]: mp1.dps = 4

In [34]: mp2.dps = 8

In [35]: mp1.cos(1)
Out[35]: mpf('0.5403061')

In [36]: mp1.cos(1) + mp1.cos(1)
Out[36]: mpf('1.080612')

In [37]: mp2.cos(1) + mp2.cos(1)
Out[37]: mpf('1.080604611')

This works because the mpf instances store a live reference to the precision of the context that created them:

In [43]: f1 = mp1.cos(1)

In [44]: f2 = mp2.cos(1)

In [45]: f1._ctxdata
Out[45]: 
[mpmath.ctx_mp_python.mpf,
 <function object.__new__(*args, **kwargs)>,
 [17, 'n']]

In [46]: f2._ctxdata
Out[46]: 
[mpmath.ctx_mp_python.mpf,
 <function object.__new__(*args, **kwargs)>,
 [30, 'n']]

In [47]: f1.context
Out[47]: <mpmath.ctx_mp.MPContext at 0x7fed42848ed0>

In [48]: f1.context is mp1
Out[48]: True

In [49]: f2.context is mp2
Out[49]: True

That can be used by methods like __add__ and __repr__ to retrieve the context precision. That is different from the way that contexts work in gmpy2 or decimal but is arguably better. The difference is that in gmpy2/decimal there is a single active global context that always affects the behaviour of operations like __add__ unless the context's add method is used like ctx.add(f1, f2).

One way this is a bit flakey though is that sometimes there will be mpfs from different contexts:

In [87]: f1*f2
Out[87]: mpf('0.29192658151797381250293427379467042520362856982479944')

In [88]: f2*f1
Out[88]: mpf('0.29192658141')

Here __mul__ takes the context from the first operand disregarding that of the second. In practice it is just not a good idea to mix the mpfs from different contexts like this. What gmpy2 and decimal provide is context methods like mp1.add and mp1.mul that can be used to explicitly control which context is being used but mpmath does not have those methods.

The docs should generally recommend using a local context like this rather than manipulating the global context. Instead of

from mpmath import mp
mp.dps = 100

it can be:

from mpmath import MPContext
mp = MPContext()
mp.dps = 100

Better would be if the FPContext accepted arguments like:

from mpmath import MPContext
mp = MPContext(dps=100)

If this were the recommended way to create the context then the confusion in gh-657 would possibly not have happened. I think this is a reasonable way to use mpmath both for interactive usage and for simple one file scripts. The only other thing I would potentially change is perhaps the name MPContext to make it something more memorable/intuitive for users who need to remember what to type when working interactively. A large fraction of users who use mpmath directly will not want more than this as far as setting the context properties goes.

In library usage it is then necessary to be able to use things like extraprec, workdps etc and these are available as methods on the context:

In [81]: mp = MPContext()

In [82]: with mp.workdps(10):
    ...:     print(mp.cos(1))
    ...: 
0.5403023059

In [83]: with mp.workdps(20):
    ...:     print(mp.cos(1))
    ...: 
0.5403023058681397174

A larger codebase using this would need to pass around this context object mp so that it can call these methods with its own context rather than the global one.

Utilities like workdps update the precision attached to the context. This is local to the context object created by mp = MPContext() so it does not affect any other code that is not explicitly using the same local variable mp. However it is in some sense global because each mpf created by mp stores a live reference to mp so changing the context has a global effect on each of those instances:

In [90]: mp = MPContext()

In [91]: f = mp.cos(1)

In [92]: f
Out[92]: mpf('0.54030230586813977')

In [93]: f._ctxdata[2]
Out[93]: [53, 'n']

In [94]: mp.dps = 30

In [95]: f
Out[95]: mpf('0.540302305868139765010482733487152')

In [96]: f._ctxdata[2]
Out[96]: [103, 'n']

In [97]: mp.cos(1)
Out[97]: mpf('0.540302305868139717400936607442955')

This is necessary for __add__ to work in conjunction with workdps:

In [101]: mp.dps = 10

In [102]: f1 = mp.cos(1)

In [103]: f1
Out[103]: mpf('0.540302305868')

In [104]: with mp.workdps(20):
     ...:     print(f1 + f1)
     ...: 
1.0806046117359073833

In [105]: f1 + f1
Out[105]: mpf('1.080604611736')

Here __add__ uses f1's context and so sees that mp's precision has changed and displays a higher precision result.

I think that probably in library usage of mpmath it would be better to have the option to avoid depending on mutating context settings altogether. Ideally it would be possible to get new contexts and call their methods like:

mp_extra = mp.extraprec_new(4)
f_add_extra = mp_extra.add(f1, f1)
f_add = mp.normalize(f_add_extra)

This way every operation comes from a particular context and no global state affects anything. This style of usage is automatically thread-safe without any need to use special thread-local storage. Passing immutable context objects around as arguments to functions also makes it straight-forward to cache the results of different operations with e.g. lru_cache. This also makes the code using mpmath easier to reason about and less bug prone.

[skirpichev]

That is different from the way that contexts work in gmpy2 or decimal but is arguably better.

I'm not sure. Because we have functions in the global mpmath namespace, that implicitly have notion of the active global context (which is actually mpmath.mp and can't be changed so far).

I don't think that banning such usage is a good idea. And also we can't "guess" the local context from arguments of such functions, because, for example, we want to feed them with python's builtin types, like mpmath.sin(1).

One way this is a bit flakey though is that sometimes there will be mpfs from different contexts...

This is a good example why the notion of the "active" context (like gmpy2's) is useful.

What gmpy2 and decimal provide is context methods like mp1.add and mp1.mul that can be used to explicitly control which context is being used but mpmath does not have those methods.

Actually, mpmath has fadd, fsub and so on, so you can explicitly control the context for arithmetic ops. But I think it will be too verbose.

Better would be if the FPContext accepted arguments like:
... potentially change is perhaps the name MPContext to make it something more memorable/intuitive for users

+1 Do you have some suggestions about naming?

Maybe we should have a single factory function to produce contexts, named (surprise!) as context() with one mandatory argument. On another hand, different context could have different options...

This is necessary for __add__ to work in conjunction with workdps:

gmpy2's way seems to me better:

>>> ctx = gmpy2.context(precision=37)
>>> f = ctx.cos(1);print(f)
0.540302305868
>>> with gmpy2.local_context(precision=70):
...     print(f+f)
... 
1.0806046117359073832631
>>> print(f+f)
1.0806046117359074

In fact, mpf looks rather as a mutable type...

Ideally it would be possible to get new contexts and call their methods like

It's possible right now, but this will be too verbose if we force users to explicitly reference context for every arithmetic operation.

it would be better to have the option to avoid depending on mutating context settings altogether.

Yeah, I see benefits of this. IIRC, the bigfloat package has immutable contexts.

In short:

1. I doubt we can abandon the notion of the "current" context, if the mpmath will have mathematical functions in its global namespace as it has now. Probably, instead we should provide some better API to get/set the current context.
2. I like the idea of immutable contexts & using constructor arguments instead of setting attributes. This is a huge API breakage, however.
3. I'm not sure that printing settings (e.g. pretty option) should be part of the context object.

[oscarbenjamin]

What gmpy2 and decimal provide is context methods like mp1.add and mp1.mul that can be used to explicitly control which context is being used but mpmath does not have those methods.

Actually, mpmath has fadd, fsub and so on, so you can explicitly control the context for arithmetic ops.

Oh, I missed those.

• I doubt we can abandon the notion of the "current" context, if the mpmath will have mathematical functions in its global namespace as it has now. Probably, instead we should provide some better API to get/set the current context.
  2. I like the idea of immutable contexts & using constructor arguments instead of setting attributes. This is a huge API breakage, however.

I am not suggesting to abandon anything or break compatibility. Adding an alternate interface for creating contexts can be done without breaking compatibility. At the same time a new interface can make other changes such as by returning an immutable context. If a new interface is considered to be better then the documentation can be changed to suggest using it.

Currently all docstrings use from mpmath import * and I think a lot of users already know that they probably don't want to use that but the docs don't suggest other ways to do anything. For the case of a user wanting to write a simple script I still don't think that the docs should suggest from mpmath import * just like the numpy docs religiously use import numpy as np everywhere. In fact the idiom of using import numpy as np is so ubiquitous that the numpy docs skip out the import and just show examples like np.cos(np.array([1, 2, 3])) and experienced Python programmers can easily recognise np.cos out of context and understand what it means. I think that mpmath should recommend a style of usage that is at least acceptable to use in the context of bigger scripts and import * is not that. Having every docstring suggest import * just means that a lot of users will try to figure out for themselves a different style of using mpmath.

So if the suggested usage is not from mpmath import * then right now it can be

from mpmath import mp
mp.dps = 50
x = mp.cos(1)

This is close enough to import numpy as np. This is not really "nicer" for the user though than

from mpmath import context
mp = context(dps=50)
x = mp.cos(1)

This version has the advantage of not depending on global state. If the context function referenced here is a new API then it does not need to preserve backwards compatibility with anything so it could be changed to return an immutable context. Since every mpf stores a reference to its context that means that everything works fine for a user writing a script that does lots of calculations but only needs a single global precision in those calculations. They can still use + rather than mp.fadd etc. This style of usage is also something that can be used by library authors because it avoids either mutating or depending on any mutable global state.

I expect that most "end users" will not want to do much more than set a single precision and do all of their calculations with that. If there is a need to extend the precision with e.g. extraprec then we would need to consider how that should work in the case of immutable contexts. Currently every mpf holds a reference to a mutable context and so extraprec mutates that context which implicitly affects all operations with any mpfs attached to that context. If we want the context to be immutable then there needs to be a way to increase the context and also upgrade the attached mpfs to the new context. That could be something like:

mp_extra, [x_extra, y_extra] = mp.extradps(4, [x, y])
z_extra = mp_extra.cos(x_extra) + mp_extra.sin(y_extra)
print(z_extra)

3. I'm not sure that printing settings (e.g. pretty option) should be part of the context object.

I agree and also __repr__ should reflect the precision that was used when the mpf was created e.g. x should print in the higher precision used to create it even after the with block:

In [31]: with mp.extraprec(50):
    ...:     x = mp.mpf(1)/3
    ...: 

In [32]: x
Out[32]: mpf('0.33333333333333333')

[skirpichev]

Adding an alternate interface for creating contexts can be done without breaking compatibility. At the same time a new interface can make other changes such as by returning an immutable context.

I think these statements are contradictory. You can't just add an additional immutable context type: after this you will have to change a lot of code to allow using this context in the mpmath (most functions now alter their context variable).

Having every docstring suggest import * just means that a lot of users will try to figure out for themselves a different style of using mpmath.

Yes, I think we should adapt docstrings and sphinx docs to avoid this, but this is slightly a different issue.

Since every mpf stores a reference to its context that means that everything works fine for a user writing a script that does lots of calculations

I thought we do agreed that this is a horrible idea, unless we make all contexts immutable (in which case this looks like gmpy2.mpfr, where numbers have precision settings). But even after this, that "guessing" of precision settings from arguments of arithmetic ops is fragile and will break few remaining algebraic identities of floating-point arithmetics, e.g. commutativity.

but only needs a single global precision in those calculations.

Explicit is better than implicit. That's why I think that this notion of a "single global precision" should be transparent for the user.

[oscarbenjamin]

You can't just add an additional immutable context type: after this you will have to change a lot of code to allow using this context in the mpmath (most functions now alter their context variable).

Many functions do not alter the context e.g. the libmp functions are all pure. For the higher level functions that do alter the context the immutable context could delegate to an internal mutable context.

Another point about immutable contexts is that creating a context should be made faster than it currently is. Currently it is slow because of all of the metaprogramming, decorators and _wrap_specfun etc that get used during MPContext.__init__. Mostly what is happening during __init__ is dynamically assigning methods that could just be declared as ordinary methods in a class statement. Making use of immutable contexts means that creating a context needs to be cheaper than a typical function call to an end-user function like mp.cos.

[skirpichev]

Many functions do not alter the context e.g. the libmp functions are all pure.

This is relatively low-level stuff. See #178 for some plans to replace mpf_/mpc_ layers with more high-level code like in functions/.

Another point about immutable contexts is that creating a context should be made faster than it currently is.

BTW, before any big change as proposed here, we, probably, should start with writing benchmarks for current high-level code.

[oscarbenjamin]

See #178 for some plans to replace mpf_/mpc_ layers with more high-level code like in functions/.

That discussion looks a little dated now. I'm not sure which part of that would still be relevant any more.

[skirpichev]

That discussion looks a little dated now.

Yes, and I thank you for your feedback in #178. Yet it seems relevant as some comment on historical code changes and it has also some notes about contexts. BTW, many old issues (til around #216, I think) are irrelevant now - I'm slowly sorting this out and I would appreciate a second view.