ENH: _StringFuncParser to get numerical functions callables from strings

[alvarosg]

This PR is part of a larger PR originally proposed to include FuncNorm and PiecewiseNorm. It was decided to split it into several PRs for easier review starting from the simpler classes.

The new functionality is inside a class _StringFuncParser in cbook, specialized on returning a _FuncInfo object with callables and other information about some predefined functions by just feeding it with a string.

A typical example of use is:

   func_info=_StringFuncParser('sqrt')
   func_info.direct #Is a callable implementing the square root
   func_info.inverse #Is a callable implementing the inverse of square root: the square

It also allows parametrical functions such as:

    _StringFuncParser('root{3}(x)')  # Receives a parameter (3) to define the degree of the root

    _StringFuncParser('log{7}(x+{4})')  # Receives a parameter (7) to define the base 
                                        # of the root, and a constant (4)

The immediate use of this will be mainly related to FuncNorm and PiecewiseNorm, but in general it may come in handy for any other function taking callables. The advantage over just passing the callable, is that we have access to more information such as the inverse function, or any other property that we may want to store, without having to ask the user directly for all of those.

For now the classes are defined as hidden, so we do not have to guarantee back compatibility in case it is decided to change the way it is designed in the future. However, this still allows other matplotlib functions/classes to use it.

[alvarosg]

@story645 @NelleV @efiring @tacaswell @QuLogic
I just wanted to point you at this, as you were probably the most involved people in the original PR.

[story645]

So this is definitely the approach I support if FuncNorm is going to support string functions, but on the other hand I wonder if string function support isn't scope creep/overkill. Like I wonder if people want to use string functions, should they be using sympy (does matplotlib support sympy) or an external funcparser library?

[story645]

is this notation common?

[alvarosg]

Yes, cbrt is used by numpy as scipy for the cubic root.

[story645]

Hmm, wondering now if there's a way to generically support any function if the name is passed in using a lookup of numpy math function names...(the implementation of the functions would also be pitched to numpy, which maybe should be happening anyway instead of lambdas...)

[QuLogic]

getattr(np, name)

[alvarosg]

But this does not really work, because it would not give have access to the inverse function, unless, we explicitly include a list of corresponding inverse functions, which will limit in any case the usage to whatever we allow.
Also it would make the parameters option very complex, because in those cases we would not control the exact shape of the string.

I think with polynomials, roots, and offset logs we can probably cover everything that basic users will want to do, for normalizations. The advanced users can always provide the callables.

[story645]

So even without getattr, I still prefer calling the numpy functions when possible because they're tested and optimized in a way a bunch of lambda functions aren't.

[alvarosg]

Sure, but this wil only be possible for the simples cases, none of the para-metrical cases would work, without the lambda. On the other hand inside the lambda the operators will be overloaded by numpy when working with arrays, so I do not think the reliability can be that different.

[story645]

shrugs I think there's potential for error/typos/etc. in writing your own and in theory the numpy functions are more robust. Also now wondering do you cast lists to arrays or are the lambda functions done in a comprehension in FuncNorm?

[alvarosg]

Yes, everything is cast into a numpy type automatically.

[story645]

no base2? (Yeah, this all feels weirdly arbitrary)

[alvarosg]

Yes, I should include log2

[story645]

not sure why you don't do a generic 'log{p}' then instead of log10 since users will have to type log10 anyway

[alvarosg]

I did it like this historically (there were no parameters at the beginning) and then keep it to in a similar way that log10, log and log2 exist separately in numpy. But you are right, being strings probably we can just keep 'log' and 'log{p}'. In any case I definitely should include 'log{p}' which is now missing.

[story645]

both log can be called log(x,2) or log(x,10) so I figure one unified call is cleaner here.

[alvarosg]

Let's decide about this later

[story645]

these should probably be properties instead of getters

[alvarosg]

I am actually considering getting rid of this, and just support get_func, as a property, and then use the FuncInfo object to obtain the required functions.

[story645]

I think if params should work equally well.

[story645]

You should raise KeyError if you're explicitly catching the KeyError, otherwise I don't see the point in differentiating the value errors.

[alvarosg]

Well, actually from the point of view of the dictionary it is a key error, but from the point of view of the user, it may be seen as a value error in one of the input arguments, as he does not need to know that this is stored in a dictionary and access with the string as a key. In any case, I am happy to switch to KeyError.

The reason I have two cases was to differentiate from TypeError, because in this case it may not be as safe to cast the input as a string for printing the error message. The way the code is now there is no chance it will not be a string, so yes, I am happy to just have a single except.

[story645]

So yeah, then I vote for use one for now and split if a need arises.

[story645]

what is this block of code doing?

[alvarosg]

The func.direct(0.5, params) outside the try block is just a debugging residue.
The one inside the block is meant to check that the callable actually works with the number of parameters parsed. Alternatively I could add a number of expected parameters to _FuncInfo, and then do a manual check on the lengths, I did not do it like this to avoid adding more stuff to the initialization of FuncInfo.

[story645]

Hmm, please remove debugging residue but I also think that's a weird non-obvious test that the callebe works with parameters passed. Like if it fails, I think it should do so naturally further down the stack where it'll yield it's own specific error message.

[alvarosg]

Sure, I actually though about this more and there is no way I could fail there, because the implementation guarantees that the parameters are floating point numbers. And the fact that there is a mathing string guarantees that the number of parameters is right.

What I am going to include is the possibility of checking if the parameters are in the right range, and also checking wheter the fucntion is bounded as fucntion of the parameters. This way, we can also allow parameters equal to 0.

[alvarosg]

@story645 The main advantage of this over other regular func parsers is that it allows access to the inverse function which is the whole point of why I needed it. I do not want to have a long discussion about FuncNorm and PiecewiseNorm on this PR, but by using this parser then the user only has to specify the direct function as a string, without the need to specify the inverse function, which is always required. If the user was to use an external parser, then he would still have to provide a callable for both, which is what I want to avoid in the simple predefined cases.

Using something like sympy could actually be used to obtain inverse functions analytically as well, however, as far as I know matplotlib does not depend on sympy, and I do not think adding a new dependency is really an option.

[story645]

My argument isn't that matplotlib should depend on sympy, but that maybe the user should be forced to define their own function and inverse (possibly using sympy) as there's a tradeoff here on code bloat and usefulness. shrugs Not something I have a strong opinion on.

[alvarosg]

Yeah, I guess that was the idea form the beginning, but then I realised, that, specially in the piecewise norm, which receives a list of functions, the code can be simplified a lot just by letting the user use the strings:

Essentially this (replace lambdas by some output from sympy if you wish):

    norm = colors.PiecewiseNorm(flist=[lambda x: x,
                                       lambda x: x**(1. / 4),
                                       lambda x: x,
                                       lambda x: x**(1. / 4),
                                       lambda x: x],
                                finvlist=[lambda x: x,
                                          lambda x: x**4,
                                          lambda x: x,
                                          lambda x: x**4,
                                          lambda x: x],
                                refpoints_cm=[0.2, 0.4, 0.6, 0.8],
                                refpoints_data=[0.2, 0.4, 0.6, 0.8])

vs

    norm = colors.PiecewiseNorm(flist=['linear', 'root{4}', 'linear',
                                       'root{4}', 'linear'],
                                refpoints_cm=[0.2, 0.4, 0.6, 0.8],
                                refpoints_data=[0.2, 0.4, 0.6, 0.8])

and the same applies to a lesser extent to FuncNorm, and MirrorPiecewiseNorm.

Of course, users who want total control will probably have to use the callables in any case, but if you just want to set a trend (See examples in previous PR), the strings are much more user friendly.

I agree that attempting to make a really powerfull func parser is not the main purpose here, but for now we can think of this as a helper class for this particular purpose, and the fact that is hidden, essentially allows us a lot of flexibility. The reason to put it in cbook, instead of colors is so it can be reused by other developers if they find it useful.

[story645]

shrugs I think there's potential for error/typos/etc. in writing your own and in theory the numpy functions are more robust. Also now wondering do you cast lists to arrays or are the lambda functions done in a comprehension in FuncNorm?

[story645]

both log can be called log(x,2) or log(x,10) so I figure one unified call is cleaner here.

[story645]

if you insist on writing your own lambdas instead of using numpy, then please test that the lambda functions are correct (no stray typos or the like)

[alvarosg]

There is a parametric test for each possible case, and it tests both the direct and the inverse, so it should be ok.

[story645]

np.power, kwargs={x2:2}
maybe something like that?

[alvarosg]

But I am not sure how would that help, you would still need the lambda function to pass the kwargs, right?

In any case, I do not think it really is that relevant at this point, we are trying to do something relatively simple for a subset of functions (for which it can be tested) aimed to help FuncNorm and children. If we ever decide to make this accessible class directly accesible to end users, or to let users create their own cuestom _FuncInfo objects, then we can worry about these things.

[story645]

depends on how it'd be called in FuncNorm...(I'd argue FuncNorm should support passing in f=np.power, kwargs={}) too since that's common enough for numpy. Would also give users a way to call np.piecewise directly.

[alvarosg]

I agree that FuncNorm should support that (even though it is not as useful as it may seem, imagine a case where the variable is the second argument and not the first one). Anyway we can discuss that in the next PR.

IMO doing that here is a mistake, as it goes against what we are trying to achieve, which is to provide the simplest possible way to specify a simple function. For more complex cases, the user will always be able to use FuncNorm with callables directly, or with a new kwargs option based on your suggestion.

[efiring]

No major problems, just some code simplifications and naming recommendations.

[efiring]

I think you can use the builtin callable here: if callable(bounded_0_1):

[alvarosg]

I think the callable built in function does not exist for python 3.0 or python 3.1, would this be a problem?

[QuLogic]

We only support 3.4+.

[efiring]

Also callable here.

[efiring]

Add comma after "parameters".

[efiring]

Naming preference: I suggest "function" and "inverse" rather than "direct" and "inverse".

[efiring]

Replace this block with is_str = isinstance(str_func, six.string_types).

[efiring]

We try hard to avoid bare except: constructs; here, use except ValueError.

[efiring]

This is potentially confusing because the error is that params[i] is not a number, not that it is zero. Just say "Parameter %i is %s, which is not a number." % (i, param)

[efiring]

Unnecessary, given checking in the init? If it is needed, it should be there.

[efiring]

Use for i, param in enumerate(params):

[efiring]

Minor tweak: formatting of the string on python 3 will be better if you replace self._funcs.keys() with [k for k in self._funcs.keys()]. This is redundant on python 2, but still works.

[QuLogic]

Or list(self._funcs) as in the cleanup PRs.

[codecov-io]

Current coverage is 62.03% (diff: 89.85%)

No coverage report found for master at 86b32ca.

Powered by Codecov. Last update 86b32ca...69f2a04

[alvarosg]

@efiring It should be done with the proposed changes. Let me know if anything else comes up, or if you want me to squash the commits.

[story645]

Who is this message for? What is check params and how would the user know if they messed this up?

[alvarosg]

This message is for whoever makes instances of _FuncInfo, which given the fact that it is underscored, would most likely be a developer. For now this only happens inside _StringFuncParser, which should have everything right, and never trigger the exception, but I wanted to make it robust enough to be used independently.

check_params is a callable that takes a list of the parameters and returns if that list of parameters is allowed for the function. For example:
'log{2}(x+{0.1})' will have its parameters extracted params=[2.,0.1], and will be transformed into 'log{a}(x+{a})'. Then the check_params stored for that parametric string function will be called check_params(params), and will return True.

If on the other hand if 'log{-2}(x+{0.1})', was called, then params=[-2.,0.1], and check_params(params) will return False, as -2 is not a valid value for the base.

I will make the message a bit more clear, as well and the unofficial documentation of the class.

[story645]

still don't think you need to bother with a log10 since you have the generic case and the user will never know (they have to write log10 anyway)

[story645]

think it should just be log and log{p} cases

[efiring]

But by far the most common use case is log10; I think that use of any other base for a norm, or for most plotting purposes, will be somewhere between rare and nonexistent. Therefore it makes sense to leave in 'log10'.

[story645]

I just think that since log10 is implicitly supported by logp, the redundancy isn't worth it unless it has significant time savings. But shrugs defer to you of coursen

[story645]

"{} must be a string".format(str_func)

[alvarosg]

I am not sure if it is a good idea to format something that we know is not a string into a string. What about simply " 'str_func' must be a string"?

[alvarosg]

I though about it again, and realized that actually your way it is better for cases when the error shows up to an user that does not know where the exception is being raised exactly, so, thank you for your suggestion, it now produces a message like that one :).

[story645]

since the else is so much shorter, what about flipplng:

if not self._params:
  func = _FuncInfo(func.function, func.inverse,
                            func.is_bounded_0_1())
else:
    #longer block of code

[story645]

you can drop the if params' 'cause when params is none, the loop just won't happen

[alvarosg]

I agree for the loop but there is also a piece of code after the loop and within the if block 'str_func = re.sub(regex, '{p}', str_func)', however I still think you are right, because if the original call to 'params = re.findall(regex, str_func)' returns no elements, then re.sub used with the same regex should not replace any elements.

[story645]

"%s is an invalid string.

[story645]

I almost feel like "Invalid check_params function" might be clearer. It'll hopefully force the user to just look up the check_params documentation, 'cause this error message is getting horribly convoluted, which yes is partially my fault.

[alvarosg]

In that case, maybe just "Invalid 'check_params' argument"? Because 'None' is also a valid value, and that way it does not necessarily implies that it has to be a function.

[story645]

Yup, that works.

[story645]

typo, "has, not "as"

[QuLogic]

Small documentation tweaks.

[QuLogic]

This should probably be done in numpydoc style.

Class used to store a function

Attributes
----------
function : function??
    The direct function
inverse : function??
    The inverse function
bounded_0_1 : bool
    ...
check_params : callable
    ...

[alvarosg]

Yeah, it is a hidden helper class for pure internal use. Most classes in these cases do not even have documentation, so I figured rules for formatting would not be as strict, so did not really put any attention to follow numpydoc... I will change it.

[QuLogic]

And why is this wrapped so early? It looks weird when the next bullet isn't.

[efiring]

I don't see how the doc build failure could be related to this. The big question is whether all of this machinery provides enough benefit to the user to be worth maintaining, and the small question is whether so many versions of log should remain as named options. We can return to both of these questions if necessary when considering the next stages of this sequence of PRs. For now, I think this is ready to go.

[efiring]

#7629 refers to the doc build problem that is unrelated to this PR. Merging also triggered a different unrelated failure in the pytest build (a pdf image comparison failure.)

[alvarosg]

@efiring Yes, every single one of the builds for the past couple of days was getting the doc build problem.

About the image comparison failure, I have already seen this kind of behaviour in the past, and most cases it was solved on itself. I was not able to determine if it was due to non deterministic behavior (i.e. plots generated with some randomly initialised data, without properly setting the seed), or it was due to some kind of crossing across versions where a newer testing image was compared to an obsolete version of the code generating it....

[alvarosg]

@efiring @QuLogic @story645 Also, quick question: Most repositories that I contribute to require squashing all commits from a PR into a single one before merging. I understand that there is a loss of information in that process, but I also see the point of not having multiple commits from a single PR in the big scale of things.
Should I squash my commits right before it is going to be merged next time?

[tacaswell]

If you feel compelled to squash, no one will complain, but in most cases we will not ask you to (if you have 100 commits, most of which are thrashing we will).

[tacaswell]

@alvarosg Thanks for your work on this 🎆