Refactor tags

[mloning]

Is your feature request related to a problem? Please describe.

• use of "capabilities" rather than "tags" in classification module
• unclear naming of _all_tags method
• inconsistent tags for composition algorithms (see Refactoring Stacking, Multiplexer, Ensembler and TransformedTarget Forecasters #977)
• integration in all_estimators function to make them usable in finding/filtering models

Describe the solution you'd like

• refactor capabilities into tags
• rename _all_tags to get_tags and make public
• add tag kwarg to all_estimators
• add tests to catch inconsistent tags

[mloning]

From #977:

A few thoughts:

• I see tags as being defined on the class (not object) level, conceptually I think of them as describing algorithm properties, so I would prefer not to update them dynamically
• In the cases of the above mentioned (composition) algorithms, for the implementation (i.e. input validation of the forecasting horizon), the tag should be set to False, the composite doesn't require the forecasting horizon during fitting, it merely passes it on to the components, which have their own tag and raise an error accordingly if they require the forecasting horizon during fitting
• For documentation, when users may use tags for finding relevant models, setting it to False may be misleading, but conceptually still makes sense, as it refers to the composition algorithm, rather than a particular instance of the composite

[fkiraly]

I see tags as being defined on the class (not object) level, conceptually I think of them as describing algorithm properties, so I would prefer not to update them dynamically

Yes, but do you see the problem? The properties of composites may depend on components.
Especially in pipelines this is important, so it would be good if there is something like a "dynamic tag".
At least, it should be settable or overridable at construction.

the tag should be set to False, the composite doesn't require the forecasting horizon during fitting, it merely passes it on to the components, which have their own tag and raise an error accordingly if they require the forecasting horizon during fitting

disagreed!

I think each forecaster should be able to tell the user in advance whether it will raise an error when no fh is passed to fit. What else is the purpose of these tags? That's also what you are testing for.

Now the stage at which the forecaster should be able to tell that is up for discussion though, i.m.o - my preference would be when constructed, i.e., as an object.

[TonyBagnall]

the problem I have with the tags in scikit learn is they are not inherited, but rather shadowed for each subclass, due to multiple inheritance. This means checking tags requires a recursive call up the inheritance hierarchy. This is hideous imo. We could use tags, but we need to be able to recover the values in the base class fit if we move to fit/_fit, so we will need inheritance and overriding to make that work, or to use this ludicrous programming pattern. I'm not even sure that dynamic binding even works properly/how I would expect it to in python, I need to check, I dont trust this language

[fkiraly]

the utility function _all_tags that @mloning wrote mimicks nested inheritance, and it's a capability that's reatively easily supplemented. This might just as easily be turned into a python getter.

[fkiraly]

where is that method

in sktime.base._base.BaseEstimator, to be refactored and moved into BaseObject in #877/#891

[TonyBagnall]

this also raises another issue, why not use scikit learn tags? I believe this code is cloned from scikit

I realise its not a strong argument, but I object to this on aesthetic grounds. If I want to query whether a classifier has a capability, I have to iterate over the whole inheritance hierarchy, build a new one off data structure of the tags, then query that data structure. I cannot buy into such horrible, inefficient design when a much simpler, clearer system is available.

[fkiraly]

If I want to query whether a classifier has a capability, I have to iterate over the whole inheritance hierarchy, build a new one off data structure of the tags, then query that data structure. I cannot buy into such horrible, inefficient design when a much simpler, clearer system is available.

what is the simpler system? This is simply inheritance of nested attributes, quite simple (but not out-of-the-box in python)

[fkiraly]

What I would propose is that we have two annotaitons:

• class tags, for the general class functionality - this is "worst case" properties
• object tags - these are created upon construction. The new get_tags method looks for class tags first, then overrides these with any object tags that were created in __init__, dependent on parameter settings.

[TonyBagnall]

if it were using inheritance normally it would not have to traverse the inheritance hierarchy, you could just query the data structure directly because it has been, you know, inherited. Also, I dont like random free form string tags, I prefer boolean variables. The simpler system (imo) is the one we currently use for classifiers, where capabilities are declared in the classifier and passed to the check function. I don't want to abandon this just yet, particularunless we are going to use the same tags as scikit learn, as it all seems a bit unnecessary to me and only justifiable for scikit compliance.

[fkiraly]

the problem is, you want the set of capabilities/tags inspectable - and for that it seems the only good solution is have them in a list/dict, as opposed to as general attributes - note that attributes can be anything, not just tags, so it is not explicitly inspectable which attributes are tags or not.

[fkiraly]

design anti-pattern: "secret" knowledge, i.e., the information is only decodable if you know undocumented conventions. get_tags or similar does not require such undocumented conventions.

[fkiraly]

So here is an interesting question, @TonyBagnall: are all sensible tags that we expect capabilities?
I.e., related to properties or scitype aspects of the estimator?

E.g., a tag that tells interface counterparts what the internal data representation used is (np.array or pd.DataFrame) is not a "capability" in a narrow sense, right?

In this case, should we have a hyper-tag that tells us whether a tag is a capability?

[fkiraly]

@mloning, @TonyBagnall, any opinions about "class tags", and "object tags" that can override "class tags" at construction?
With the override happening in _all_tags.

[TonyBagnall]

I'm not that sure how static variables work in python, but I believe our capabilities are static. However, we are not necessarily using them as static variables! I'll have a look

https://stackoverflow.com/questions/68645/are-static-class-variables-possible-in-python

personally, I prefer boolean variables to free form string tags, but thats probably just me

[fkiraly]

I'm not that sure how static variables work in python, but I believe our capabilities are static. However, we are not necessarily using them as static variables

The problem is that the capabilities ("scientific" properties) are not necessarily static, they may depend on hyper-parameters and components.
This is rare for "primitive type" hyper-parameters, but the typical case for complex hyper-parameters or estimator components.

So in my opinion we need a way to set, or at least override, them at construction and perhaps even at any parameter setting that happens (the latter may be overkill).

Regarding boolean variables: I think we need more, since some tags are a way to get scityping.
See, e.g., PR #980 and #1013 which I feel would be unnecessarily complicated by insisting on boolean tags - we would have to laboriously "decode" multiple boolean tags into one of multiple types or strings.

[TonyBagnall]

I'm not militant about this, so if its changed then I wont complain...

However, my view is that capabilities relate to the functionality of the class. So if say unequal_length == true, an object of this class is capable of taking as input unequal class values and is safe to build with data having that characteristic. There may be many ways a class can deal with a data characteristic, and there may be an object variable that can determine that and be set in the constructor. However, the capabilities are simply for error checking: if you pass a certain type of data and it will crash the algorithm, it should throw a meaningful exception. Configuring capabilities doesn't make sense to me, that should require new code. Having capabilities declared at the class value makes it very easy to see what a class can do. However, it may screw up the _fit/fit model assuming the point of that is to generalise the checks before overloaded _fit() is called, I'm not sure how python handles this kind of polymorphism.

[fkiraly]

So if say unequal_length == true, an object of this class is capable of taking as input unequal class values and is safe to build with data having that characteristic

Yes, but please consider what I'm saying.

Whether unequal class values can be dealt with an object, may depend on the parameters and components of that object. Consider a pipeline with transformers and a classifier at the end. An object of the "pipeline" type will be able to deal with that data if and only if the classifier at the end can.

This (logically) shows that the (scientific/logical) property cannot be a property of the class, but is in fact a property of the object with given parameters.

From a user/query perspective, one may want to distinguish "always" and "sometimes" as a class property.

[mloning]

@fkiraly could you write down what you think the main reasons are for having object-level tags (e.g. tags updated dynamically depending on components) rather than static class-level tags? I think is related to how we want to use tags.

[fkiraly]

discussion notes from hackmd today:

• tags discussion - Franz, Markus, Martin, Taiwo, Tony
  • use of tags - semantic/indexing lookup for user, or only internal/testing?
    • importance in checks and conversions (e.g., what to)
    • input validity & related properties
    • algorithmic features/properties, availability of interface points e.g., can produce performance estimates
    • writing common tests for estimators with the same tags
    • user guidance
  • inheritance and default values
  • if inheritance: child classes to specify all tags, or only tags deviating from default? (FK: perhaps all, more robust against changes to defaults)
    • inheritance yes, and override only non-defaults
    • tests for robustness
  • object or class level
    • both for long-term; short/mid-term, focus on class level
    • object and class level should both be inspectable
    • user wants to be shown object level one typically
  • documentation of tags - where, how?
    • fixed description of tags, but where?
    • ALL_TAGS to be factored out and supplemented by plain english descriptions; docs auto-generated from that
  • how do we agree on which tags?
    • that file has codeowners, it's us
  • "meta-tags", e.g., which tags apply to which scitypes?
    • dataframe with three columns? tag name, list of scitypes, plain English description?
    • maybe structured strings in tags, like "forecaster:supports_exogeneous_X"; some might be generic, like "handles_missing_data" or "multivariate" (?)
  • forecaster tags - boolean?
    • can have non-boolean, but we need to carefully watch testing
  • which tags should we have? for forecasters?
    • [ENH] Define forecaster tags #957
  • lookup of tags, all_tags like all_estimators? by scitype?
    • yes, and is easy with the table above
  • lookup of estimators based on tags?
    • ML: yes should be easy to implement via an additional filtering of the list of collected estimators
  • display of estimators with tags, Summary of code features for sktime classifiers #995, Summary of the Capabilities of time series classifiers #996
    • great idea, should be autogenerated from above
  • tag refactor? default tags in BaseForecaster; added some new tags #1013

[mloning]

Notes from our discussion today:

contributors: @fkiraly, @mloning, @aiwalter, @thayeylolu, @TonyBagnall

• use of tags - semantic/indexing lookup for user, or only internal/testing?
  • importance in checks and conversions (e.g., what to)
  • input validity & related properties
  • algorithmic features/properties, availability of interface points e.g., can produce performance estimates
  • writing common tests for estimators with the same tags
  • user guidance
• inheritance and default values
• if inheritance: child classes to specify all tags, or only tags deviating from default? (FK: perhaps all, more robust against changes to defaults)
  • inheritance yes, and override only non-defaults
  • tests for robustness
• object or class level
  • both for long-term; short/mid-term, focus on class level
  • object and class level should both be inspectable
  • user wants to be shown object level one typically
• documentation of tags - where, how?
  • fixed description of tags, but where?
  • ALL_TAGS to be factored out and supplemented by plain english descriptions; docs auto-generated from that
• how do we agree on which tags?
  • that file has codeowners, it's us
• "meta-tags", e.g., which tags apply to which scitypes?
  • dataframe with three columns? tag name, list of scitypes, plain English description?
  • maybe structured strings in tags, like "forecaster:supports_exogeneous_X"; some might be generic, like "handles_missing_data" or "multivariate" (?)
• forecaster tags - boolean?
  • can have non-boolean, but we need to carefully watch testing
• which tags should we have? for forecasters?
  • [ENH] Define forecaster tags #957
• lookup of tags, all_tags like all_estimators? by scitype?
  • yes, and is easy with the table above
• lookup of estimators based on tags?
  • ML: yes should be easy to implement via an additional filtering of the list of collected estimators
• display of estimators with tags, Summary of code features for sktime classifiers #995, Summary of the Capabilities of time series classifiers #996
  • great idea, should be autogenerated from above
• tag refactor? default tags in BaseForecaster; added some new tags #1013

[mloning]

From our discussion today (@fkiraly and @mloning):

• we want to implement both dynamic and static tags
• which public interface points do we want:
  • get_tags - retrieves static, overridden by dynamic
  • get_tag(key, default=None)
  • set_tags(**tags)
  • mirror_tags(estimator, which=all) (i.e. `set_tags(estimator.get_tags()[which])
  • maybe also get_static_tags(), this is called in get_tags(), currently static tags via _all_tags()
  • instance variable _dynamic_tags, dict
• dynamic tags need to be set for all composite estimators in constructor, specific for compositions, but may be optional (only if tags are affected by component properties)
• agreed with FK proposal - todo is renaming of _all_tags to get_static_tags
• do we want to set/update tags every time we retrieve them (current implementation), or update them and store them only when needed (using some cache)?
  • for dynamic tags cache would have to be updated at the beginning of fit (as components may change via set_params)
  • _set_dynamic_tag(key, value)
• keep set_params in mind but start with constructor only in 1st iteration
• we need unit testing
  • interface logic testing - one mock class or object (on BaseObjects)
    • dynamic tags: inheritance: four tags, override by child class yes/no, override by instance yes/no get_tag and get_tags should retrieve the right thing
    • get_tag default setting works
    • set_tags setting a new and existing tag, read out the dict directly
    • static tags: get_tags() on class == _tags class attribute
    • get tag on obj == obj._dynamic_tags[tag]
  • individual class/objects: inherits from BaseObject?

Partly implemented in #1099.

[fkiraly]

@mloning, @RNKuhns, the above is now implemented in #1134. I made a new PR due to extensive changes downstream - in contrast to #1099, this is no longer downwards compatible with internal interfaces due to changes in method names and removal of internal interface point _all_tags and external interface point _has_tag. It should still be downwards compatible with external interfaces.

[fkiraly]

I called the class version get_class_tag(s) rather than get_static_tag(s) since that seemed a more "obvious" name. Happy to change it back though.

[TonyBagnall]

regarding tags vs capabilities of classifiers: there is a difference. Capabilities relate to the underlying nature of the problem, not just the data structure used to store it. So, for example, unequal length series and equal length series are both stored in nested pandas. Missing values can be in arrays and pandas. Whether the algorithm is able to handle such data is a capability, and how it copes with it cannot be generalised to the base class.

So, for example, if we make the shapelet transform classifier capable of handling unequal length series, then if it is passed a panda it should convert it to np.array if the problem is equal length, but leave as pandas if not. Other classifiers may want to pad unqual or truncate by default, but there is no generic best approach. I would not want to pad by default, since the user may be unaware of what is going on. It is better to throw an exception and get them to use a transformer if they want to pad in the general case.

Example 2, an interval based classifier is passed series with missing values. By default, it should throw an exception, unless it has a specific capability to deal with it. For example, if averaging over an interval, it could just ignore missing unless the wwhole interval is missing, then just use 0.

[fkiraly]

@TonyBagnall, but that's exacly what tags are meant for!

I.e., "can deal with missing values", "can handel unequal length time series", are prime examples of tags!

I was actually thinking to name tags like that "capability:unequal_length_panel", or "capability:missing_values" or similar.

Note that we usually attach tags to individual algorithms, not base classes.

[fkiraly]

Here are some example of current tags, which are very similar to those, and even includes the one with missing values:

    (
        "univariate-only",  # todo: rename to "scitype:handles_exogeneous"
        "forecaster",
        "bool",
        "does forecaster use exogeneous data (X)?",
    ),
    (
        "handles-missing-data",
        "estimator",
        "bool",
        "can the estimator handle missing data (NA, np.nan) in inputs?",
    ),
    (
        "scitype:y",
        "forecaster",
        ("str", ["univariate", "multivariate", "both"]),
        "which series type does the forecaster support? multivariate means >1 vars",
    ),

[TonyBagnall]

I'm conflating this with data conversion, I forgot about the bespoke inheritance model,, but you will end up testing twice if a data frame has unequal length and missing, in fit and _fit, unless the data stores the meta data, or having horribly complex tags/bespoke base class attributes

[fkiraly]

but you will end up testing twice if a data frame has unequal length and missing, in fit and _fit,

Hm, why would we end up testing twice?

I think all the checking of capabilities against user queries should be done in fit.
In _fit you can assume that any data arriving and any query arriving is in line with the estimator's capabilities.
I.e., if not capable of handling missing values and unequal length, the _fit can rest assured that the arriving data container has no missing values and has equal length time series.

See for instance #1195 for an implementation example around prediction intervals.
The user can, post-PR, assume that return_pred_int=False when working in _fit, whenever the tag capability:pred_int=False.

[TonyBagnall]

It arrives in a compatible format, but the algorithm needs to switch on characteristics not format. So if it has missing values the algorithm is different to when it does not.

So I pass an array with missing values. Tag says classifier can handle it, so tests pass in fit, data does not change. We go to _fit, then we need to know if there are missing before fitting, so need to check again.

[fkiraly]

@TonyBagnall, I don't really get your conceptual model here, sorry.

For instance, what does the capability have to do with the _fit? If the algorithm can handle missing values, wouldn't you assume there are some, in _fit, and there is just one version? Where the "no missing values" version is a degenerate special case?

Maybe good to have that "tags" meeting finally.

[fkiraly]

Answering your post from #1195 here since it´s not really about prediction intervals but tags:

it seems reasonable, but I still think we will end up checking in both fit() and _fit() unless the data characteristics are stored. So lets say we have missing values, and a classifier can handle this internally. in fit() we check_X, requires a pass through the whole data set to check, but its all fine so it passes. We then call _fit(). How does _fit know that the data has or doesnt have missing values? We could store somewhere that it contains missing values as a result of the call in _fit, but that seems a bit tangled and brittle imo.

So really my point is about the interaction between _fit(), fit(), checkX and using the information from checkX. It would all be solved if the data were tagged ....

I think you need to check twice anyway:

• once, whether the data set contains NA
• second, where precisely the NA are

The first is typically much faster. The second you need to do to actually do something with the NA, and is less fast. Further, the real bottleneck is typically not the check but the algorithm.

Now, a "smart" data container knows whether there are NA in them, and pandas does this a bit halfway, e.g., you know whether dtypes in columns are nullable.

[TonyBagnall]

if you know there are no missing you do not need to check whether every element is a missing value in _fit, making it more efficient. The bigger issue though is unequal length series, which effects the algorithm more fundamentally.

All of this is solved if we get checkX to return a hash table of the data characteristics stored in the base class.

[fkiraly]

All of this is solved if we get checkX to return a hash table of the data characteristics stored in the base class.

That's an interesting idea - sth like infer_schema(self, X)?
Then, data properties can be compared to capability tags?

I'd see this separate from the tagging issue though, since it's related to data properties and possibly the data container itself.
For instance, a geomstats like data container could infer the data schema and properties at wrapping, and provide it as an inspectable part of the wrapper. Quick question to @ninamiolane: are you already doing sth like that?

[TonyBagnall]

tagging, splitting to fit, _fit and finding out the data characteristics are all linked in my mind, as is the separation between checking and conversion

[fkiraly]

one point that came up in #1367, tag names should be valid variable names, or they cannot be used "properly" in set_tags. This in particular impacts the larger number of tags that have a dash somewhere, which is (problematically) parsed as a minus if using them as kwarg variable names.

[mloning]

Can this issue be closed?

[TonyBagnall]

I think so, some refactoring required but perhaps contributors can raise an issue when someone finds a requirement for it or feels inclined to do the work?