Support batch-mode queries?

[dataframing]

Hi,

I've run into a bit of a use-case that I'm not sure is quite supported by modAL – nor the broader libraries for active learning – but would be relatively simple to implement. After reviewing modAL's internals a bit, I don't think it officially supports active learning with batch-mode queries.

The sampling strategies (for example, uncertainty sampling) do support the n_instances parameter, but from what I can tell, uncertainty sampling may return redundant/sub-optimal queries if we return more than one instance from the unlabeled set. This is a bit prohibitive in settings where we'd like to ask an active learner to return multiple (if not all) examples from the unlabeled set/pool, and the computational cost for re-training an active learning model goes without saying.

I found requests for batch-mode support in the popular libact library (issues #57 and #89) but, to the best of my knowledge, I'm not sure they were addressed in any of their PRs.

In that case, does it make sense to implement something like [Ranked batch-mode active learning] by Cardoso et al.? I took a crack at it this weekend for a better personal understanding, but if it's worth integrating and supporting in modAL I'm happy to polish it and talk it through in a PR.

Thanks!

[cosmic-cortex]

Hi there!

You are right, multiple instances can be selected using n_instances in the built-in query strategies, but so far, it is very simple and only returns the instances with largest utility value. It can happen that they are very close to each other in the feature space. I think it is a good idea to implement more sophisticated strategies as you suggested.

Currently, this is how a query strategy looks like in general.

def custom_query_strategy(classifier, X, a_keyword_argument=42):
    # measure the utility of each instance in the pool
    utility = utility_measure(classifier, X)

    # select the indices of the instances to be queried
    query_idx = select_instances(utility)

    # return the indices and the instances
    return query_idx, X[query_idx]

In the built-in queries, the select_instances() function is multi_argmax(values, n_instances=1) from modAL.utils.selection, which does what I have described earlier. By replacing this with the function described in the Ranked batch-mode paper, it can be included in the current query strategies easily.

What I would suggest is to add these functions to the modAL.utils.selection. If you open a pull request as you suggested, we can work together on integrating these features!

[dataframing]

Sounds good! I'll try and get a PR in by tomorrow night (my time, so Wednesday morning your time?). Thanks!

[dataframing]

Hey Tivadar, a quick question: as it turns out, I'm having a bit of difficulty integrating my implementation of the ranked batch mode learner with modAL's architecture. Not because I don't understand the individual pieces (I think I understand those...) but I think it's because it's a model and not just a query strategy...does that make sense? E.g. the paper describes being able to access the "model" training data, which doesn't seem accessible from the scope of any particular sampling function within modAL.utils.selection, unless I'm mistaken.

For what it's worth, feel free to check out my implementation in the gist here (the actual implementation is below the notebook). It's not properly commented nor done, but maybe you get an idea of what I mean? The RankedBatchLearner's query method (again, I know it's not aligned with the modAL API/architecture) relies on being able to build off of the initial training set for building the ranked batch.

[cosmic-cortex]

Thanks, great work! I'll review your implementation in detail soon.

The problem you mention about training data not accessible in the scope of functions within modAL.utils.selection is not necessarily true. The first argument of every query strategy is the active learner itself which has access to the training data. Since the selection functions are usually called within the query strategy, you can pass these as arguments for the function selecting the instances. I'll try to outline this in detail in code today after I finished reading the paper.

Also, a quick note. In modAL.density, you can find the similarize_distance decorator, which can be used for similar purposes you used the euclidean_sim. Feel free to use it if you find it suitable!

[dataframing]

Ah! I see now. If that's the case, it shouldn't be too bad going forward — I had assumed the classifier was the core estimator.

Also, a quick note. In modAL.density, you can find the similarize_distance decorator, which can be used for similar purposes you used the euclidean_sim. Feel free to use it if you find it suitable!

Hahaha yes, I hand-rolled my own during the implementation but realized it's already there and modular enough for any distance function afterwards. Thanks!!

[cosmic-cortex]

Ok, now the feature is implented and merged to the dev branch! Thanks @dataframing!

[nikolay-bushkov]

Thanks for the feature! I went through the tutorial and it works fine, but I see here https://github.com/cosmic-cortex/modAL/blob/308af9b0ffff30597431ffac5ca44e3ad518c607/examples/ranked_batch_mode.py#L36
a possibility to get X_raw.shape[0] as a training index that will result in IndexError...
Check my suspicion, please)

Also, in order to have consistent documentation across web-site, Jupyter notebooks and py-files it is convinient to use Sphinx with a couple of extensions... PyTorch tutorials, for example, are built using these things.

[cosmic-cortex]

I plan to switch to Sphinx, several people suggested that the API reference and the website should be merged. It might take a while however, since I am not familiar with it.

[nikolay-bushkov]

Recently, I was responsible for documentation refactoring here http://docs.deeppavlov.ai/en/master/intro/hello_bot.html
I can contribute to modAL doing similar refactoring if you ok with it, but at first, we need to discuss several things.

I see some sphinx/readthedocs artifacts in the repo, but eventually, Github Pages is used for the site, am I right?

I strongly suggest choosing NumPy or Google style for docstrings (http://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html). Conversion could be done with https://github.com/dadadel/pyment. For example, sklearn uses NumPy style, while PyTorch uses Google style. As for me, the latter is preferred in case of type annotation in function signatures. What do you think?

Considering type annotation. As I understand, Python 2 support is not planned (and its great!), so it is useful to force type annotation, which simplifies docstrings (type annotation is more about syntax, while docstrings is more about semantic).

Also, I have not found licensing information. I suggest using Apache 2.0 as a friendly one for both academia and industry.

Waiting for your reply...

[cosmic-cortex]

Sorry for not answering sooner, I was on vacation for the last two weeks.

It sounds great! I would really appreciate your help! I have opened up a new issue (#22) for discussing this. Github pages is used for the site itself and readthedocs for the autogenerated documentation. I would like to merge these two just as the PyTorch docs for instance, as you have mentioned.

I am not really familiar with the NumPy or Google style themselves, but I'll take a look ASAP.

Regarding licensing, I use MIT license (https://github.com/cosmic-cortex/modAL/blob/master/LICENSE).