Add `MABSelector` operator selection class to allow using MABWiser to select operators #153

P-bibs · 2023-05-16T14:23:21Z

This PR:

Closes Integrate with MABWiser library #132.
Adds and passes relevant tests.
Has well-formatted code and documentation.

Changelog

Adds a new MABSelector class that extends the OperatorSelectionScheme class. Library users can use MABSelector to wrap a multi-armed-bandit object from MABWiser and use it as an operator selection scheme
The new MABSelector mirrors the existing operator selection API, including support for op_coupling.
The MABSelector includes an optional argument context_extractor that takes a function with the signature State -> np.array. If a contextual multi-armed-bandit algorithm is used, this argument must be supplied and will be used to extract a context vector from your domain-specific state object.

Example Usage

import mabwiser
from mabwiser.mab import MAB 

policy = mabwiser.mab.LearningPolicy.EpsilonGreedy(0.1)                           
mab = MAB(                                                                         
        learning_policy=policy,
        arms=MABSelector.make_arms(num_d_operators, num_r_operators))              
select = MABSelector(                                                              
        [6, 4, 1, 0.5],            
        mab,                       
        num_d_operators,           
        num_r_operators,           
        )                          
alns.iterate(initial_solution, select, accept, stop)

TODO

Pass formatting and linting checks
Add doc strings to new functions
How to support contextual MABs
Add example of MABSelector usage
~~Bump version in pyproject.toml?~~
~~rewrite git history to include better messages (or squash)~~
Write tests that test the output of operator selection

* add docstrings to all MABSelector methods * update MABSelector API: instead of taking a MAB and rebuilding it internally with the correct arm names, expose a function that the MABSelector creator calls to create the arm names as we expect them * properly respect op_coupling in MABSelector

* add tests for MABSelector, still need to add tests for prediction functionality * add validation statements at the beginning of static methods on MABSelector

P-bibs · 2023-05-16T14:27:25Z

A few comments right off the bat:

How do we want to handle adding an example usage of the MABSelector. It's a small enough tweak versus other selectors that I don't know if it warrants a full-sized example like the notebooks in the examples directory. Would it make sense to add to the "ALNS Features" example?
How to handle contextual MABs. The trouble I see with forcing all States that are used with the MABSelector to conform to a ContextualState protocol is that not all of the MABs are contextual, and as far as I know MABWiser does not expose an API to see if the currently selected policy requires context.
How do you want to handle git history? Do you usually squash or should I rewrite some of the sloppier commit messages?
It seems building the docs is failing although not because of a change I made? I got the html docs to build locally so not sure what could be causing the problem.
@N-Wouda @skadio

N-Wouda · 2023-05-16T18:08:10Z

@P-bibs hi, thanks for the PR! I'll go through the code changes in a minute. First, in response to various parts of the accompanying text:

How to support contextual MABs

I think we can demand this via a protocol class, e.g. ContextualState. Something like (see also the ALNS State):

from typing import Protocol

class ContextualState(Protocol):
  def objective(self) -> float: ...
  def get_context(self) -> np.ndarray: ...

(I'm guessing the context is a numpy vector)

Add example of MABSelector usage

I think this can get some real estate in the features notebook. Later on we can perhaps add another example where we explicitly compare different selection schemes, but for now I'd go for what's simplest to implement (and that is probably adding it to the features notebook).

Bump version in pyproject.toml?

No need, I only do this when I make a new release to PyPI. Before then we live in sin for a little while on master, but I'm OK with that :-).

rewrite git history to include better messages (or squash)

I'll squash merge the PR at the end, so this is all good.

How to handle contextual MABs. The trouble I see with forcing all States that are used with the MABSelector to conform to a ContextualState protocol is that not all of the MABs are contextual, and as far as I know MABWiser does not expose an API to see if the currently selected policy requires context.

I think we can get away with demanding context in general for the MABSelector. The type checker wouldn't mind as long as we do things properly in the tests, and end users can always opt not to satisfy the type hint if they know better (e.g., they use a context-free policy and know what they're doing). All we can do is hint that maybe a user should think about providing context, we don't need to enforce it, I think.

It seems building the docs is failing although not because of a change I made? I got the html docs to build locally so not sure what could be causing the problem.

I just fixed that and merged it into this branch. Should be good now!

codecov · 2023-05-16T18:19:03Z

Codecov Report

Merging #153 (9b41fb2) into master (6bb423d) will decrease coverage by 0.70%.
The diff coverage is 90.74%.

@@             Coverage Diff             @@
##            master     #153      +/-   ##
===========================================
- Coverage   100.00%   99.30%   -0.70%     
===========================================
  Files           28       29       +1     
  Lines          665      719      +54     
===========================================
+ Hits           665      714      +49     
- Misses           0        5       +5

Impacted Files	Coverage Δ
alns/select/MABSelector.py	`89.79% <89.79%> (ø)`
alns/State.py	`100.00% <100.00%> (ø)`
alns/select/__init__.py	`100.00% <100.00%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

skadio

Done with my first pass -- this is a lot less code and cleaner integration than I thought it would be! Good job.

There are some last mile items, comments inline. The most high-level ones being readme polishing + usage example + notebook with contextual.

One learning question to @N-Wouda: how did you decide to use "pairs" of destroy, repairs as the operating mechanism --as opposed to keeping the two separate?

Tuples makes it A LOT easier to cast the problem as a Bandit --each tuple becomes an arm. But I am wondering, if they were separate; we could have one Bandit to select over destroy arms and another Bandit to select over repair arms. I am curious if this is just a "syntactical" difference OR there is something more to it in terms of richer structure? faster reward/learning? more expressive/representation power?

README.md

alns/select/MABSelector.py

skadio · 2023-05-16T18:17:35Z

alns/select/MABSelector.py

+            self._mab.fit(
+                [self._operators_to_arm(d_idx, r_idx)],
+                [self._scores[outcome]],
+                contexts=self._context_extractor(cand),


So when we call mab.fit(), if the context_extractor returns None, and things work fine..

@P-bibs BTW, mab objects has a "is_contextual" field. I assume that can come handy

skadio · 2023-05-16T18:20:55Z

README.md

+
+The mabwiser integration isn't fully documented yet, but you can get started by looking at the example in `examples/tsp_mab.py`. The only difference from the provided `examples/travelling_salesman_problem.ipynb` is the following lines
+```
+# USE A MAB ALGORITHM AS A SELECTOR


We need to make this usage example a bit more concrete. We can only focus on the non-contextual version here in the Readme and assume the standard ALNS parts, like operators, but flash the connection between MAB creation and ALNS calling.

Ideally, it would be great to add a notebook under /examples, say mabwiser_alns.py that has at least two things:

the full example from readme that actually runs and compiles

a contextual version (does not have to super meaningful)

N-Wouda

This already looks really good, thank you @P-bibs! In my comments I've mostly focused on the high-level interface. I'm not sure if everything I wrote below is possible, so please let me know if something doesn't work the way I thought it did. Definitely let me know if you have a better idea for some of these things!

README.md

N-Wouda · 2023-05-16T18:17:48Z

pyproject.toml

@@ -33,6 +33,7 @@ classifiers = [
 python = "^3.8, <4.0"
 numpy = ">=1.18.0"
 matplotlib = ">=3.5.0"
+mabwiser = "^2.7.0"


Can we add this as an extra? Something like:

[tool.poetry.dependencies] python = "^3.8, <4.0" numpy = ">=1.18.0" matplotlib = ">=3.5.0" mabwiser = { version = ">=2.7.0", optional = true } [tool.poetry.extras] mabwiser= ["mabwiser"] # optional integration with MABWiser

This way, pip install alns will only install the existing core dependencies. pip install alns[mabwiser] then also installs mabwiser. We can test whether mabwiser is installed in MABSelector.py.

I've made the needed changes to pyproject.toml and also tried my hand at adding code to raise an exception in MABSelector.py (right before def __init__(...). Can you take a look and make sure this is what you had in mind?

Sure! I'm about to hop on a train so I should have some time in a few hours.

N-Wouda · 2023-05-16T18:32:19Z

alns/select/MABSelector.py

+        def extract_context(state):
+            if context_extractor is None:
+                return None
+            else:
+                context = context_extractor(state)
+                if isinstance(context, list):
+                    # if the output is a list, wrap it so it's 2D. Otherwise,
+                    # it's an np array or dataframe, which can be left alone.
+                    context = [context]
+                return context
+
+        self._context_extractor = extract_context


With the protocol assumption on ContextualState, I think we can replace calls of the form self._context_extractor(state) with state.get_context(). If MABWiser expects a 2D argument, we can cast the return value to something 2D as np.atleast_2d(state.get_context()).

alns/select/MABSelector.py

N-Wouda · 2023-05-16T19:15:06Z

alns/select/tests/test_mab.py

+
+
+# TODO:
+# tests that check the predictions of the mab


I think such tests will be really valuable! You might have a look at how we test AlphaUCB for inspiration.

N-Wouda · 2023-05-16T19:32:21Z

How to handle contextual MABs. The trouble I see with forcing all States that are used with the MABSelector to conform to a ContextualState protocol is that not all of the MABs are contextual, and as far as I know MABWiser does not expose an API to see if the currently selected policy requires context.

I think we can get away with demanding context in general for the MABSelector. The type checker wouldn't mind as long as we do things properly in the tests, and end users can always opt not to satisfy the type hint if they know better (e.g., they use a context-free policy and know what they're doing). All we can do is hint that maybe a user should think about providing context, we don't need to enforce it, I think.

If I understand @skadio correctly, we can test whether context is needed using the is_contextual field. Let's do that: if is_contextual is true, we attempt to extract a context vector from the state (via state.get_context()). If it's false, we do not provide a context and also do not call state.get_context(). Then a user is only bothered for a context when they really have to provide one, and is otherwise left alone.

N-Wouda · 2023-05-16T19:59:12Z

One learning question to @N-Wouda: how did you decide to use "pairs" of destroy, repairs as the operating mechanism --as opposed to keeping the two separate?

Tuples makes it A LOT easier to cast the problem as a Bandit --each tuple becomes an arm. But I am wondering, if they were separate; we could have one Bandit to select over destroy arms and another Bandit to select over repair arms. I am curious if this is just a "syntactical" difference OR there is something more to it in terms of richer structure? faster reward/learning? more expressive/representation power?

We do a bit of both. The older, simpler selection schemes like the roulette wheel mechanisms do exactly what you describe: they first select a destroy operator, and then a repair operator that works well with the selected destroy operator. That works well enough, so it's definitely not a terrible idea.

In the context of bandits, the tuple "action space" made more sense to me. This mostly has to do with the rewards. A new candidate solution is produced by the destroy and the repair operator working together, not separately. Assigning the reward for that candidate solution to the operator pair, rather than to each separately, seems to me very logical. Since I want to assign rewards to these operator pairs, the operator pairs naturally have to be the action, rather than each operator separately.

Hope that helps!

skadio · 2023-05-16T20:19:11Z

I overlooked the reward attribution problem. If we split, then we need to somehow attribute the reward back -- a motivation to keep them together

P-bibs · 2023-05-18T14:12:56Z

Hi all, just a quick update here: I've been away since Tuesday but will be back later today or tomorrow and will implement the fixes then (apologies for the delay!). I'll go through the PR and reply to anything I'm unsure on but other than that consider all proposed changes accepted.

* Gracefully handle missing mabwiser dependency

N-Wouda

The approach with flags is exactly what I had in mind!

N-Wouda · 2023-05-20T22:00:56Z

alns/select/MABSelector.py

+from typing import Callable, List, Optional, Tuple, Union
+
+import numpy as np
+import pandas as pd


Pandas is not an ALNS dependency - that's why the CI fails. Can we get by not typing that as a possible return value? (open question, I don't know if that's OK for MABWiser)

Could we move this into the conditional import block?

try: from mabwiser.mab import ... + import pandas as pd except: ...

Yes, I just checked: pandas is a dependency for MABWiser. Let's do what you suggest!

alns/select/MABSelector.py

N-Wouda · 2023-05-20T22:03:17Z

alns/select/MABSelector.py

+           Int. J. Artif. Intell. Tools, 30(4), 2150021:1–2150021:19.
+    """
+
+    if not MABWISER_AVAILABLE:


I think this needs to go in the init method, not in the class itself. I could be wrong, but I think this gets executed by the parser even when the user is not using MABSelector.

Darn, I spent a bit puzzling through this and thought this approach might work, but you're correct. The problem with putting this in the constructor is that the type annotations on the constructor arguments get resolved before the function body runs, and this resolution will fail if the user doesn't have MABWiser installed. Could we get away with stringifying the constructor arguments?

def __init__( ... - learning_policy: LearningPolicy, + learning_policy: "LearningPolicy", ... )

It's unclear to me whether this still results in type checking with current mypy versions.

I think that's OK for us!

Pushed a fix. I tested with the following three scenarios:

Don't have mabwiser installed and don't try to use the MABSelector (works)

Don't have mabwiser installed and try to use the MABSelector (raises an exception)

Have mabwiser installed and try to use the MABSelector (works)

P-bibs · 2023-05-23T22:56:42Z

I've fixed the conditional import and added some more tests. Serdar and I had previously discussed testing that ALNS's AlphaUCB and MABWiser's UCB1 return the same results since they are implementing the same algorithm, but as far as I can tell this isn't possible in practice since ALNS and MABWiser differ slightly on how they initialize the per-arm mean for each arm.

Only one thing left from my perspective: add support for contexts and corresponding tests.

N-Wouda · 2023-06-02T12:41:33Z

Hi @P-bibs, I'm a bit under the weather but I'll try to have a look at this sometime this weekend.

N-Wouda · 2023-06-04T09:35:33Z

@N-Wouda There's one thing I'm still not sure about re: contexts that's reflected in my most recent (unworking) commit. We had previously discussed requiring states that we use with the MABSelector to conform to the ContextualState protocol. This allows us to call get_context() to produce a context from a state. However, the selector class is not parameterized by some value with a state (this is one level higher up in the ALNS class), so it's not clear to me how we would require that states conform to ContextualState when using MABSelector but not when using other selectors.

Do you have thoughts on how to solve this? In terms of raw functionality, the current state of code reflects how I think it would work. It's just the types we need to figure out.

I think it's OK to assume a get_context() method is present on the state when using the MABSelector. So the typing in MABSelector can then use ContextualState. When the user isn't using a context-aware MAB, they should just return None from their get_context() method. That's slightly inconvenient for users, but I think it's reasonable to ask users to be aware of this the moment they want to use the interface to MABWiser.

We should explain this clearly in the docs, both for the API (MABSelector), and in an example notebook. Once we do I think we can get away with this minor 'gotcha'.

Does that help?

P-bibs · 2023-06-05T13:00:30Z

Unfortunately, I think this may still not quite work. If we change the MABSelector as follows:

-def __call__(self, rnd_state: RandomState, best: State, curr: State) -> Tuple[int, int]:
+def __call__(self, rnd_state: RandomState, best: ContextualState, curr: State) -> Tuple[int, int]:

We will get a type-check error since the OperatorSelectionScheme that MABSelector inherits from requires a State, not a ContextualState.

I will try to come up with some alternate solutions but if you have any ideas please let me know. We can always revert to the API where the user passes in the get_context function as a parameter to the MABSelector.

N-Wouda · 2023-06-05T19:24:35Z

I figured the typing would be an issue here. Good mypy recognises that! I'm OK with the types not quite working out here, and would suggest we add a # type: ignore[override] comment at the end of that line.

We could come up with something that type checks nicely, but I feel it might not be worth our time. For a first version I'd rather add a few more tests and make the documentation really clear on what we're doing here and why.

If you have something really cool that does work please feel free to showcase that. But if you don't have anything yet, I am not too bothered by this and suggest we spend our time on things that are more valuable in the short run (tests and documentation).

P-bibs · 2023-06-05T20:25:31Z

Sounds good. I will implement that change and make a longer notebook example that fleshes out the usage and motivation behind the MABSelector.

P-bibs · 2023-06-07T14:06:55Z

@N-Wouda I've added tests for contextual mabs as well as a section in the "ALNS Features" example that explains them. From my end, everything is checked off. Would you be able to give a final once-over when you have a chance?

P-bibs · 2023-06-07T14:09:03Z

alns/select/MABSelector.py

-        try:
+        if self._mab._is_initial_fit:
            prediction = self._mab.predict(
                contexts=self._context_extractor(curr)
            )
            return arm2ops(prediction)
-        except Exception:
+        else:
            # This can happen when the MAB object has not yet been fit on any
            # observations. In that case we return any feasible operator index


I had to change this to use a conditional instead of exceptions for the control flow. Since the underlying exception raised by MABWiser is simply Exception and not a specific subclass, the original try/except block was catching any exception that was thrown while calling is_initial_fit, which is probably not what we want. (an alternative would be going back to the try/except method and checking the specific exception string)

N-Wouda · 2023-06-07T20:44:22Z

I'll have a look at this tomorrow afternoon. Thanks for your hard work so far!

N-Wouda

Looks good to me! A few small things and then I think we can merge this in! 🚀

alns/State.py

alns/select/MABSelector.py

alns/select/tests/test_mab.py

pyproject.toml

N-Wouda · 2023-06-08T18:45:14Z

After those last few things are done (they should be pretty small) I'll merge this in. I'm happy with how it looks. It'll be a valuable addition to ALNS, and I'll try to have a new release out sometime next week!

P-bibs · 2023-06-09T02:25:59Z

Great! Thank you for all your help and taking time to review the changes. It sounds like there may be some research interns working with the library over the summer so we'll hear if there are any major issues soon :)

skadio · 2023-06-09T18:02:41Z

This is great news!! Thank you @P-bibs and @N-Wouda --cool collaboration!

The summer interns are going to test this out immediately once available for installation over PyPi.

They should do sth like pip install alns[mabwiser] ?

N-Wouda · 2023-06-10T08:04:17Z

The summer interns are going to test this out immediately once available for installation over PyPi.

They should do sth like pip install alns[mabwiser] ?

Yes, that's the plan! I'm going to merge this in now, check everything locally, and then release a new version to PyPI later today.

N-Wouda · 2023-06-10T09:39:37Z

alns == 5.2.0 is on PyPI now and includes the MABSelector. I have updated the documentation a bit to clarify mabwiser is not installed by default, but should be installed separately using pip install alns[mabwiser]. If a user forgets to do this they'll get an error saying the same thing, so the end result should be OK either way.

Well done @P-bibs and @skadio! Very nice collaborating on this. And @P-bibs, best of luck with your PhD!

skadio · 2023-06-10T09:49:01Z

Fantastic news for the weekend! I will share with team members here, and they can do a first-round user testing next week.

Congrats @P-bibs on your very successful senior thesis! Enjoy your summer ahead of the grad school :)

skadio · 2023-06-10T09:54:49Z

@N-Wouda very minor comment -- the comments you added in the readme of ALNS (which is great!), is it possible to bundle them under a (sub) header, say # ALNS with MABWiser or sth..

I would like to add a (sub) header in the MABWiser readme, sth like MABWiser for ALNS and then link to the ALNS library and link to "that" (sub)header.

That way, we will complete the full circle between both libs to reference users from both sides.

BTW, once the team here uses it, we might also consider a social post to spread the word. Similarly, we can talk about it in upcoming events as small extended abstracts or tool sessions etc. something to keep in mind for next conferences

N-Wouda · 2023-06-10T10:04:38Z

@skadio does https://github.com/N-Wouda/ALNS#installing-alns provide what you're looking for?

BTW, once the team here uses it, we might also consider a social post to spread the word. Similarly, we can talk about it in upcoming events as small extended abstracts or tool sessions etc. something to keep in mind for next conferences

I agree. Let's stay in touch about this over the summer. I'm also curious to hear more about the work of your summer interns!

skadio · 2023-06-10T10:09:28Z

Yup, perfect! I can link to that header in mabwiser readme (adding a section on ALNS now)

Let me circle back with findings from using the new functionality.

N-Wouda · 2023-06-10T10:19:15Z

@skadio great!

skadio · 2023-06-14T08:04:14Z

@N-Wouda @P-bibs Quick circle back after more tests.

I asked the team to do an external evaluation of our new functionality and to try out more complex MABWiser settings etc.

Good news --everything is working expected! And the team was very happy/impressed with the clean integration. It opens the door for many interesting applications/research.

The only comment that came, which I thought is worth sharing is:

In the quick start example: https://alns.readthedocs.io/en/latest/setup/template.html

Potentially, we can add the get_context() method in the ProblemState class and add a TODO comment sth like "implement if contextual mabwiser bandit is used".

That would make the context requirement more explicit and the example more complete. Not a big dial, just a comment to share.

N-Wouda · 2023-06-14T11:11:01Z

@skadio great, thanks for letting us know. I have just added get_context to the quickstart, with a short note. I hope this is a bit clearer!

P-bibs added 9 commits May 16, 2023 09:42

Add mabwiser as pyproject.toml dep

976a32e

Remove mabwiser as an optional dependency

b183bbe

Add example and rough readme description

25f39a2

Fix mab tsp example

f712f06

Update docstrings for mabselector

d7e5328

Add MABSelector to docs

0b146f4

Add tests and extra validation to MABSelector

eb3f195

* add tests for MABSelector, still need to add tests for prediction functionality * add validation statements at the beginning of static methods on MABSelector

Remove old example of solving TSP with MABSelector

88cb09e

Merge branch 'master' into master

dbbff28

skadio reviewed May 16, 2023

View reviewed changes

N-Wouda reviewed May 16, 2023

View reviewed changes

P-bibs added 6 commits May 20, 2023 14:09

Revert readme changes

4c16003

Fix typos/clarify prose in docstrings

40320db

Make op2arm methods functions instead

1414556

Remove unnecessary _primed flag

5f5ff7b

Remove make_arms, instead forward args to MAB

16fbe82

Make MABWiser an optional dependency

397e9d3

* Gracefully handle missing mabwiser dependency

N-Wouda reviewed May 20, 2023

View reviewed changes

P-bibs added 2 commits May 23, 2023 15:04

Add tests for output of MABSelector

808800b

Fix conditional import for mabwiser

a324072

Add MABSelector to "ALNS Features" example

aeefa4c

P-bibs added 3 commits June 6, 2023 10:15

Add finalized API for contextual states

4c24d6d

Add contextual bandit example to example notebook

00ef41e

Add tests for contextual MABs

577125e

P-bibs commented Jun 7, 2023

View reviewed changes

Add mabwiser as a required dep for development

692d16b

N-Wouda self-requested a review June 7, 2023 21:26

N-Wouda reviewed Jun 8, 2023

View reviewed changes

P-bibs added 2 commits June 8, 2023 22:23

Add fixes from review

cd2f73c

Update test to check for new exception

9b41fb2

N-Wouda approved these changes Jun 9, 2023

View reviewed changes

N-Wouda merged commit 963080d into N-Wouda:master Jun 10, 2023
6 of 8 checks passed

Add MABSelector operator selection class to allow using MABWiser to select operators #153

Add MABSelector operator selection class to allow using MABWiser to select operators #153

Conversation

P-bibs commented May 16, 2023 • edited

Changelog

Example Usage

TODO

P-bibs commented May 16, 2023 • edited

N-Wouda commented May 16, 2023

codecov bot commented May 16, 2023 • edited

Codecov Report

skadio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

N-Wouda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

N-Wouda commented May 16, 2023

N-Wouda commented May 16, 2023

skadio commented May 16, 2023

P-bibs commented May 18, 2023

N-Wouda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

P-bibs commented May 23, 2023

N-Wouda commented Jun 2, 2023

N-Wouda commented Jun 4, 2023

P-bibs commented Jun 5, 2023 • edited

N-Wouda commented Jun 5, 2023 • edited

P-bibs commented Jun 5, 2023

P-bibs commented Jun 7, 2023

Choose a reason for hiding this comment

N-Wouda commented Jun 7, 2023

N-Wouda left a comment

Choose a reason for hiding this comment

N-Wouda commented Jun 8, 2023

P-bibs commented Jun 9, 2023

skadio commented Jun 9, 2023

N-Wouda commented Jun 10, 2023

N-Wouda commented Jun 10, 2023

skadio commented Jun 10, 2023

skadio commented Jun 10, 2023

N-Wouda commented Jun 10, 2023

skadio commented Jun 10, 2023

N-Wouda commented Jun 10, 2023

skadio commented Jun 14, 2023

N-Wouda commented Jun 14, 2023

Add `MABSelector` operator selection class to allow using MABWiser to select operators #153

Add `MABSelector` operator selection class to allow using MABWiser to select operators #153

P-bibs commented May 16, 2023 •

edited

P-bibs commented May 16, 2023 •

edited

codecov bot commented May 16, 2023 •

edited

P-bibs commented Jun 5, 2023 •

edited

N-Wouda commented Jun 5, 2023 •

edited