Add interactive configuration generation #409

desilinguist · 2020-03-19T14:23:36Z

This PR closes #354 and #397.

It adds interactive configuration generation for all of the tools. Non-interactive or batch-mode configuration generation was previously added by #398.

To try out interactive generation, just run X generate --interactive where X is any of the five RSM command line tools. You can also add --subgroups for X = rsmtool, rsmeval, and rsmcompare.

The actual interaction bit is mostly handled by the excellent prompt_toolkit library. Here's what this this PR does specifically:

The configuration generation (both non-interactive and interactive) is handled by a class called rsmtool.utils.commandline.ConfigurationGenerator. The generate() method handles the non-interactive generation and the interact() method handles the interactive generation.
All of the required fields for the 5 tools and some of the more important optional fields are displayed to the user to provide inputs for them. The appropriate auto-completion and value validation are attached to each fields so that the user is shown appropriate options just like in interactive IPython sessions (which is where prompt_toolkit was spun out of). All of this is handled by the new class rsmtool.utils.commandline.InteractiveField. Essentially, when the interact() method is called, an InteractiveField instance is created for each of the chosen fields with the appropriate completer and validator attached and shown to the user. The values input by the user and then used to create a configuration dictionary and then eventually output as a string.
The chosen fields are contained in rsmtool.utils.constants.INTERACTIVE_MODE_METADATA which also defines the metadata for each field. This metadata is used to structure the interaction model (what to display, how to complete, how to validate etc.). The metadata consists of:
- the label to be shown for the field in interactive mode,
- the type of the field
- whether the field takes a single value or multiple values (think "subgroups")
- whether the field takes its value from a fixed list of choices
Testing for this specific functionality was quite tricky. We don't really want to test the prompt_toolkit functionality as that is already tested outside of our code. So, in keeping with the best practices of unit testing, we use mocking to mock the functionality provided by prompt_toolkit and only test that our code is doing the right thing. The new testing classes added here are test_utils.TestInteractiveField and test_utils.TestInteractiveGenerate. The first class checks that InteractiveField works as expected and the second checks that the ConfigurationGenerator.interact() works as expected. Between the two of them, we are covering the whole functionality.
This PR also adds documentation as follows: it adds a main page at the top level called "Auto-generating configuration files" that contains documentation for interactive, non-interactive generation (which was previously in each tool's configuration file page), and also API usage. I also added multiple screencasts that illustrate how the interactive generation works. Finally, I added links to this top-level page from the configuration file pages of all 5 tools and also all 5 tutorials. I updated the API documentation to include the documentation for the ConfigurationGenerator class and its generate() method and pointed to that from the top-level page too.
This PR also removes all imports from __init__.py other than the 5 main API functions. This meant minor changes to the API documentation.

- Hook this up to a `--quiet` / `-q` flag for each command line tool. - Add new tests.

- Create new `InteractiveField` class - Add docstrings to all methods.

- this makes it easy to test the class

pep8speaks · 2020-03-19T14:23:53Z

Hello @desilinguist! Thanks for updating this PR.

In the file rsmtool/utils/commandline.py:

Line 322:101: E501 line too long (119 > 100 characters)
Line 511:101: E501 line too long (128 > 100 characters)
Line 730:101: E501 line too long (115 > 100 characters)
Line 736:101: E501 line too long (108 > 100 characters)

In the file rsmtool/utils/constants.py:

Line 224:101: E501 line too long (103 > 100 characters)
Line 229:101: E501 line too long (111 > 100 characters)
Line 231:101: E501 line too long (103 > 100 characters)
Line 241:101: E501 line too long (149 > 100 characters)
Line 243:101: E501 line too long (108 > 100 characters)
Line 245:101: E501 line too long (116 > 100 characters)
Line 246:101: E501 line too long (102 > 100 characters)
Line 247:101: E501 line too long (101 > 100 characters)
Line 251:101: E501 line too long (101 > 100 characters)
Line 253:101: E501 line too long (101 > 100 characters)
Line 256:101: E501 line too long (104 > 100 characters)
Line 258:101: E501 line too long (101 > 100 characters)
Line 260:101: E501 line too long (105 > 100 characters)
Line 264:101: E501 line too long (101 > 100 characters)
Line 271:101: E501 line too long (105 > 100 characters)

Comment last updated at 2020-03-27 16:07:56 UTC

# Conflicts: # rsmtool/rsmcompare.py # rsmtool/rsmeval.py # rsmtool/rsmpredict.py # rsmtool/rsmsummarize.py # rsmtool/rsmtool.py # rsmtool/utils/commandline.py # tests/test_utils.py

coveralls · 2020-03-19T15:39:16Z

Coverage increased (+0.07%) to 90.409% when pulling 97f6493 on interactive-config-generation into ddb0452 on master.

- update the default value logic appropriately

aloukina · 2020-03-24T13:27:03Z

When generating file for RSMTool, you get some of the optional fields (e.g. second human score, min, max). In the generated file though these are listed under // OPTIONAL: replace default values below based on your data. with some fields having indeed default values but other fields having the values the user had set in the interactive mode. Is there a way to regroup the fields to keep the unassigned fields together?

aloukina · 2020-03-24T13:31:16Z

rsmeval generate displays the following prompt: Name of column in evaluation data that contains human scores, if any: - this is not an optional field so it should be Name of column in evaluation data that contains human scores with sc1 as default, right?

desilinguist · 2020-03-24T13:43:02Z

rsmeval generate displays the following prompt: Name of column in evaluation data that contains human scores, if any: - this is not an optional field so it should be Name of column in evaluation data that contains human scores with sc1 as default, right?

human_score_column is listed in the optional section for rsmeval. See here: https://github.com/EducationalTestingService/rsmtool/blob/master/rsmtool/utils/constants.py#L136

aloukina · 2020-03-24T13:44:52Z

That's because it defaults to sc1... Our definition of optional is not very transparent as I am realizing.

desilinguist · 2020-03-24T13:46:31Z

Ah, I see. Yes "sc1" will be automatically in the config file so you are saying I just need to display "sc1" in the interactive prompt.

aloukina

Few minor things as I was looking at the code.

rsmtool/utils/constants.py

desilinguist · 2020-03-24T18:53:01Z

When generating file for RSMTool, you get some of the optional fields (e.g. second human score, min, max). In the generated file though these are listed under // OPTIONAL: replace default values below based on your data. with some fields having indeed default values but other fields having the values the user had set in the interactive mode. Is there a way to regroup the fields to keep the unassigned fields together?

This may not be possible because all of the optional fields are generated in sorted order right now and the comment is inserted before the first optional field. Doing this would require extra bookkeeping - perhaps we can just tweak the message in the comment?

aloukina · 2020-03-24T18:54:31Z

I agree that it's a nice to have - let's see how much people use the functionality.

- Also link to the configuration file sections for each tool and to the tutorials.

mulhod

This looks great! By and large, my comments are superficial. I have a lot of suggestions for things like typos, etc.

doc/automated_configuration.rst

rsmtool/utils/commandline.py

Co-Authored-By: Matt Mulholland <mulhodm@gmail.com>

- Fix how field types are quoted in docstrings. - Spell out input and intermediate file extensions in docstrings.

desilinguist · 2020-03-27T00:47:09Z

Thanks for the comprehensive review @mulhod! Much appreciated. I have addressed all of your comments.

mulhod

Looks great! This is a really nice functionality. I think it will help a lot of people who might want to use RSMTool but may not know where to start.

aloukina

Looks great to me - very exciting!

doc/automated_configuration.rst

Co-Authored-By: Anastassia Loukina <aloukina@users.noreply.github.com>

…ingService/interactive-config-generation Add interactive configuration generation

desilinguist added 11 commits March 13, 2020 15:46

Add ability to suppress warnings for generate_configuration().

93be826

- Hook this up to a `--quiet` / `-q` flag for each command line tool. - Add new tests.

A bit more refactoring of rsmtool.utils package

a20066b

A first working version of the interactive generator.

45f88f5

Handle fields that are lists.

6df5b70

make code better refactored

4860724

Major refactor of interactive generation code

23b8f85

- Create new `InteractiveField` class - Add docstrings to all methods.

rename 'prompt' to 'label` in metadata directory

413dfcf

Pass in the metadata dictionary to the class

7cf00cd

- this makes it easy to test the class

Import only the main 5 API functions into the main rsmtool namespace

a4d8199

Update expected configuration JSON files

b9feaa0

Update test_utils.py

46babfb

desilinguist requested review from mulhod and aloukina March 19, 2020 14:23

desilinguist added this to In progress in RSMTool 8 via automation Mar 19, 2020

Merge branch 'master' into interactive-config-generation

cedd585

# Conflicts: # rsmtool/rsmcompare.py # rsmtool/rsmeval.py # rsmtool/rsmpredict.py # rsmtool/rsmsummarize.py # rsmtool/rsmtool.py # rsmtool/utils/commandline.py # tests/test_utils.py

desilinguist added 7 commits March 19, 2020 11:52

Shorten some very long lines.

49a6603

Reorder if statement alphabetically by type

8d02d6c

Use all supported input extensions

753ded0

Improve the ID validator

d0f3539

Allow zero entries for multiple count fields

a28c7cd

- update the default value logic appropriately

Move statement up to handle EOFError correctly

bedf69d

Add initial set of tests for InteractiveField

a252337

aloukina reviewed Mar 24, 2020

View reviewed changes

desilinguist added 5 commits March 24, 2020 10:20

Fix pep8 warnings.

7c81e6c

Fix silly typo.

688bad4

Merge branch 'master' into interactive-config-generation

abb4f01

Fix typo in comment.

83f1fc4

Incorporate PR suggestions.

8d9e8b8

desilinguist changed the title ~~[WIP] Add interactive configuration generation~~ Add interactive configuration generation Mar 24, 2020

desilinguist requested a review from aloukina March 24, 2020 15:57

Update API

d9db47f

desilinguist linked an issue Mar 24, 2020 that may be closed by this pull request

Trim imports in __init__.py #397

Closed

desilinguist added 3 commits March 24, 2020 15:02

Add a first pass at documentation for auto-generation.

84c5787

Finish documentation for automated generation

a1694cf

- Also link to the configuration file sections for each tool and to the tutorials.

Add another section for API use

0f37435

mulhod requested changes Mar 26, 2020

View reviewed changes

RSMTool 8 automation moved this from In progress to Review in progress Mar 26, 2020

desilinguist and others added 4 commits March 26, 2020 19:49

Apply suggestions from code review

3a73f26

Co-Authored-By: Matt Mulholland <mulhodm@gmail.com>

Add another test for choice field

54cc558

Fix quotation style and spell out extensions

9ddd815

- Fix how field types are quoted in docstrings. - Spell out input and intermediate file extensions in docstrings.

Change code type to text.

a146841

RSMTool 8 automation moved this from Review in progress to Reviewer approved Mar 27, 2020

mulhod approved these changes Mar 27, 2020

View reviewed changes

aloukina approved these changes Mar 27, 2020

View reviewed changes

doc/automated_configuration.rst Show resolved Hide resolved

doc/automated_configuration.rst Outdated Show resolved Hide resolved

Update doc/automated_configuration.rst

97f6493

Co-Authored-By: Anastassia Loukina <aloukina@users.noreply.github.com>

desilinguist merged commit 2faea99 into master Mar 27, 2020

RSMTool 8 automation moved this from Reviewer approved to Done Mar 27, 2020

delete-merged-branch bot deleted the interactive-config-generation branch March 27, 2020 17:37

desilinguist mentioned this pull request Mar 27, 2020

Add config generator #354

Closed

srhrshr pushed a commit to srhrshr/rsmtool that referenced this pull request Oct 23, 2021

Merge pull request EducationalTestingService#409 from EducationalTest…

a9fce2b

…ingService/interactive-config-generation Add interactive configuration generation

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add interactive configuration generation #409

Add interactive configuration generation #409

desilinguist commented Mar 19, 2020 •

edited

pep8speaks commented Mar 19, 2020 •

edited

coveralls commented Mar 19, 2020 •

edited

aloukina commented Mar 24, 2020

aloukina commented Mar 24, 2020

desilinguist commented Mar 24, 2020

aloukina commented Mar 24, 2020

desilinguist commented Mar 24, 2020

aloukina left a comment

desilinguist commented Mar 24, 2020

aloukina commented Mar 24, 2020

mulhod left a comment

desilinguist commented Mar 27, 2020 •

edited

mulhod left a comment

aloukina left a comment

Add interactive configuration generation #409

Add interactive configuration generation #409

Conversation

desilinguist commented Mar 19, 2020 • edited

pep8speaks commented Mar 19, 2020 • edited

Comment last updated at 2020-03-27 16:07:56 UTC

coveralls commented Mar 19, 2020 • edited

aloukina commented Mar 24, 2020

aloukina commented Mar 24, 2020

desilinguist commented Mar 24, 2020

aloukina commented Mar 24, 2020

desilinguist commented Mar 24, 2020

aloukina left a comment

Choose a reason for hiding this comment

desilinguist commented Mar 24, 2020

aloukina commented Mar 24, 2020

mulhod left a comment

Choose a reason for hiding this comment

desilinguist commented Mar 27, 2020 • edited

mulhod left a comment

Choose a reason for hiding this comment

aloukina left a comment

Choose a reason for hiding this comment

desilinguist commented Mar 19, 2020 •

edited

pep8speaks commented Mar 19, 2020 •

edited

coveralls commented Mar 19, 2020 •

edited

desilinguist commented Mar 27, 2020 •

edited