Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #6322: Add IsProperSubsetOf to item-selection interaction #6350

Merged
merged 85 commits into from Mar 27, 2019
Merged
Show file tree
Hide file tree
Changes from 81 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
53f0db1
added ng-cloak to prevent DEV-MODE from showing up. ng-cloak prevents…
Buahahahah Feb 23, 2019
6f67904
fixed dev mode showing up, by binding it
Buahahahah Feb 23, 2019
3c16759
fixed linting issues
Buahahahah Feb 23, 2019
2e930f8
fixed linting issues(1)
Buahahahah Feb 23, 2019
2fc32cb
fixed linting issues(1)
Buahahahah Feb 23, 2019
53c3dc7
Delete docstrings_checker
amey-kudari Feb 23, 2019
9e56386
Delete re
amey-kudari Feb 23, 2019
9935d05
Delete base.html
amey-kudari Feb 23, 2019
e2bff3c
Revert "Delete base.html"
Buahahahah Feb 23, 2019
7dc63ef
deleted extra files
Buahahahah Feb 23, 2019
9ca34f4
Merge https://github.com/oppia/oppia into fix6322
Buahahahah Feb 25, 2019
6fbd579
added option - IsProperSubsetOf, to Item-selection
Buahahahah Feb 25, 2019
202e4b2
fixed linting errors
Buahahahah Feb 25, 2019
7eab79a
fixed linting errors
Buahahahah Feb 25, 2019
8281ecc
fixed karma tests
Buahahahah Feb 25, 2019
a5b9d39
undo previous pull request
amey-kudari Feb 25, 2019
ebbd172
remove changes
amey-kudari Feb 25, 2019
69d96e5
try fix, add feedback 4 option
Buahahahah Feb 26, 2019
242f959
try fix, wrong addition
Buahahahah Feb 26, 2019
d8b5a2a
Add issue template for server errors (#6308)
nithusha21 Feb 23, 2019
ba9fff5
added ng-cloak to prevent DEV-MODE from showing up. ng-cloak prevents…
Buahahahah Feb 23, 2019
2accfe9
fixed dev mode showing up, by binding it
Buahahahah Feb 23, 2019
3889e6b
fixed linting issues
Buahahahah Feb 23, 2019
535f968
fixed linting issues(1)
Buahahahah Feb 23, 2019
096bda1
fixed linting issues(1)
Buahahahah Feb 23, 2019
1b73f79
Add a script for updating indexes. Move gcloud functions into a separ…
seanlip Feb 23, 2019
031f4b7
Delete docstrings_checker
amey-kudari Feb 23, 2019
bf2191c
Delete re
amey-kudari Feb 23, 2019
dd9d991
Delete base.html
amey-kudari Feb 23, 2019
37e2468
Revert "Delete base.html"
Buahahahah Feb 23, 2019
f81de7d
deleted extra files
Buahahahah Feb 23, 2019
8fe4738
Fix #2306: Fixes signup issue when a new tab is opened (#6235)
bansalnitish Feb 24, 2019
b71d0c8
Port back-end tests from Travis to CircleCI (#6234)
apb7 Feb 24, 2019
3406256
Fix part of #6254: Refactor for directives.js and filters.js (#6275)
ankita240796 Feb 24, 2019
bcb1f35
added option - IsProperSubsetOf, to Item-selection
Buahahahah Feb 25, 2019
59fac23
fixed linting errors
Buahahahah Feb 25, 2019
3936da7
fixed linting errors
Buahahahah Feb 25, 2019
30ef871
fixed karma tests
Buahahahah Feb 25, 2019
d6efd32
undo previous pull request
amey-kudari Feb 25, 2019
5210a32
remove changes
amey-kudari Feb 25, 2019
abe9585
try fix, add feedback 4 option
Buahahahah Feb 26, 2019
524d1ec
try fix, wrong addition
Buahahahah Feb 26, 2019
c1a2bee
Merge branch 'fix6322' of https://github.com/amey-kudari/oppia into f…
Buahahahah Feb 26, 2019
9f65674
Fixed one error
Buahahahah Feb 27, 2019
e3976ec
Remove extra use of feedback_4 with no coresponding outcome
Buahahahah Feb 27, 2019
32ab85e
add function and do relevent modifications
Buahahahah Feb 28, 2019
dad139d
fixed frontend errors
Buahahahah Feb 28, 2019
8a83e28
Merge branch 'fix6322' of https://github.com/amey-kudari/oppia into f…
Buahahahah Feb 28, 2019
a6a2448
Merge branch 'develop' into fix6322
Buahahahah Feb 28, 2019
2603e4f
change copyright details
amey-kudari Feb 28, 2019
e005656
Explain function IsProperSubsetOf
amey-kudari Feb 28, 2019
5d333a5
Added test for isProperSubsetOf()
amey-kudari Feb 28, 2019
fc6c718
Correct coding style
amey-kudari Feb 28, 2019
9f4b615
correct typo
amey-kudari Feb 28, 2019
fdd76ef
fixed typos in comments
Buahahahah Mar 1, 2019
93c62ae
fixed typos in all_interactions.yaml
Buahahahah Mar 1, 2019
106f366
update demo test case
Buahahahah Mar 2, 2019
5e048f5
update demo test case-2
Buahahahah Mar 2, 2019
72c00af
corrected the Input-validation for IsProperSubsetOf
Buahahahah Mar 4, 2019
099b1b4
fixed lint errors
Buahahahah Mar 4, 2019
14e4ff7
added tests for inputValidation for function IsProperSubsetOf
Buahahahah Mar 5, 2019
97e1271
corrected error messages, and removed console logs
Buahahahah Mar 5, 2019
0d73e7a
corrected lint errors
Buahahahah Mar 5, 2019
00c7334
corrected lint errors
Buahahahah Mar 5, 2019
21579af
Grammer
Buahahahah Mar 6, 2019
66086a1
add comments
Buahahahah Mar 6, 2019
14eb83d
corrected grammer, and fixed StateResponseDirective
Buahahahah Mar 7, 2019
5b0993b
Validation service, validate IsProperSubsetOf in all cases
Buahahahah Mar 7, 2019
ed45178
remove all console statements used in debugging
Buahahahah Mar 7, 2019
8c92410
fix lint errors
Buahahahah Mar 7, 2019
2d61b9a
corrected description
Buahahahah Mar 7, 2019
efe39c7
remove the extra for loop that runs only one time
Buahahahah Mar 8, 2019
524e0ba
add the for loop
Buahahahah Mar 9, 2019
1bebe87
removed extra code, corrected grammer
Buahahahah Mar 9, 2019
af3984c
fixed typos and added test cases
Buahahahah Mar 10, 2019
cf48203
removed extra code that didnt modify anything
Buahahahah Mar 10, 2019
f5dd3e7
added a >1 checking and did not extend areAllOptionsCovered
Buahahahah Mar 12, 2019
6cc668a
fix lint issues
Buahahahah Mar 12, 2019
4125b98
fixed brace style
Buahahahah Mar 12, 2019
b8c8454
corrected validation file and removed double space
Buahahahah Mar 15, 2019
7054f4a
removed the else condition and created a seperate check for ProperSub…
Buahahahah Mar 16, 2019
59e1772
push added code block for 'isProperSubsetOf' down
Buahahahah Mar 17, 2019
4e052b2
push added code block for 'isProperSubsetOf' down
Buahahahah Mar 17, 2019
0465d4f
push areAllChoicesCovered back into brackets
Buahahahah Mar 17, 2019
5b6c66c
Merge branch 'develop' into fix6322
Buahahahah Mar 22, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
18 changes: 18 additions & 0 deletions data/explorations/all_interactions/all_interactions.yaml
Expand Up @@ -578,6 +578,7 @@ states:
feedback_1: {}
feedback_2: {}
feedback_3: {}
feedback_4: {}
hint_1: {}
interaction:
answer_groups:
Expand Down Expand Up @@ -629,6 +630,23 @@ states:
rule_type: DoesNotContainAtLeastOneOf
tagged_misconception_id: null
training_data: []
- outcome:
dest: Item Selection Multi
feedback:
content_id: feedback_4
html: <p>That's good! No bad options, but make sure to choose all good options!</p>
labelled_as_correct: false
missing_prerequisite_skill_id: null
param_changes: []
refresher_exploration_id: null
rule_specs:
- inputs:
x:
- <p>Good option A.</p>
- <p>Good option C.</p>
rule_type: IsProperSubsetOf
tagged_misconception_id: null
training_data: []
confirmed_unclassified_answers: []
customization_args:
choices:
Expand Down
Expand Up @@ -41,6 +41,16 @@ oppia.factory('itemSelectionInputRulesService', ['$filter', function($filter) {
return normalizedInput.some(function(val) {
return normalizedAnswer.indexOf(val) === -1;
});
},
// This function checks if the answer
// given by the user is a subset of the correct answers.
IsProperSubsetOf: function(answer, inputs) {
var normalizedAnswer = $filter('removeDuplicatesInArray')(answer);
var normalizedInput = $filter('removeDuplicatesInArray')(inputs.x);
return normalizedAnswer.length < normalizedInput.length &&
normalizedAnswer.every(function(val) {
return normalizedInput.indexOf(val) !== -1;
});
}
};
}]);
Expand Up @@ -72,4 +72,20 @@ describe('Item Selection rules service', function() {
['a', 'b', 'c', 'd'], RULE_INPUT)).toBe(false);
}
);
it('should have a correct \'is a proper subset of\' rule',
function() {
var RULE_INPUT = {
x: ['a', 'b', 'c']
};
expect(isirs.IsProperSubsetOf([], RULE_INPUT)).toBe(true);
expect(isirs.IsProperSubsetOf(['a'], RULE_INPUT)).toBe(true);
expect(isirs.IsProperSubsetOf(['a', 'b'], RULE_INPUT)).toBe(true);
expect(isirs.IsProperSubsetOf(['a', 'b', 'c'], RULE_INPUT)).toBe(false);
expect(isirs.IsProperSubsetOf(['d', 'e'], RULE_INPUT)).toBe(false);
expect(isirs.IsProperSubsetOf(
['a', 'b', 'c', 'd'], RULE_INPUT)).toBe(false);
expect(isirs.IsProperSubsetOf(
['b', 'c', 'd', 'e'], RULE_INPUT)).toBe(false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add one more test to ensure empty set is also considered as a proper subset.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bansalnitish , Done, Thanks for reviewing!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and also one more case where the answer is not in the subset at all.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amey-kudari did you miss this comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seanlip I added this test

expect(isirs.IsProperSubsetOf(['d', 'e'], RULE_INPUT)).toBe(false);

in line 84

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but you need to reply to the comment too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, sorry I will make sure I reply to all reviews individually (I replied in the end saying l did all of them, #6350 (comment))

}
);
});
Expand Up @@ -131,6 +131,8 @@ oppia.factory('ItemSelectionInputValidationService', [
'please select only one answer choice.')
});
}
} else if (rule.type === 'IsProperSubsetOf') {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only the case if the rule's input contains >= 2 objects, right? ['a'] is not a proper subset of ['a'].

Given that this error slipped through, I suggest that you add Karma tests for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seanlip, I am sorry I did not understand the error here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To modify code, you'll need to start by understanding it. Can you explain, in your own words and in detail, what (a) the getAllWarnings() function is doing, and (b) what this block of code (from line 111 onwards) is doing, conceptually?

Once you've done that I can talk you through the error but I need to make sure we have a common baseline first.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how I had understood it,
I am not sure what customizationArgs is,
In handeledAnswer, all indices in seenChoices is initialized to false
from line 111, if the author set max number of options that can be selected as 1,
Then we iterate over answerGroup,
here I am a little confused,
I think answer group.rules says the type of operation, and for each operation, we check if the given combination is correct or not. and it modifies the index of what it checked to be true or leave it as false

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, thanks for explaining. I think you might need to read more of the codebase to understand what customizationArgs and answerGroups are. Your onboarding mentor @ankita240796, or other folks, might be able to point you in the right direction.

Also /cc @vibhor98, who is an expert on interactions and should be able to help. Please ask him if you have specific questions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, Thanks @seanlip, I will read through them, and ask @ankita240796 and @vibhor98 in case I get stuck trying to understand it

handledAnswers[choiceIndex] = true;
} else if (rule.type === 'ContainsAtLeastOneOf') {
handledAnswers[choiceIndex] = true;
} else if (rule.type ===
Expand All @@ -144,10 +146,30 @@ oppia.factory('ItemSelectionInputValidationService', [
});
});
});
areAllChoicesCovered = handledAnswers.every(function(handledAnswer) {
return handledAnswer;
});
}
answerGroups.forEach(function(answerGroup, answerIndex) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This for-each loop is redundant. You can add if (rule.type === 'IsProperSubsetOf') condition in the above code only. In general, please try avoiding unnecessary duplicate code whenever possible. Sorry for not realising it earlier! Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @vibhor98
Yeah, Ill remove it, I was going to remove it, but later saw that the for loop runs only one time, and instead of using ruleinputs[0], I thought Ill keep it the same way, any1 else trying to modify it later, wont need to go through it again. Ill remove it now.
Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The recent changes are wrong. It looks like you haven't understood the code properly yet. I meant moving this if condition entirely inside the for-each loop of line no: 121 (where all other rules are present) and there is no need of this redundant for-each loop.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! @vibhor98 , I am not sure, but based on what I understood,
The for loop, I removed by mistake thinking something else, Ill add it again
Also ,
That for each loop (line 121

ruleInputs.forEach(function(ruleInput) { 

is inside an if loop,

 if (maxAllowedCount === 1) {

And this warning does not need maxAllowedCount to be 1.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, yes! I think I missed maxAllowedCount customisation arg. In that case, please add the loop that you've removed by mistake and everything else should be fine then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, added it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done,

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would put this after the "areAllChoicesCovered" bit. That bit relates to the computation that was previously done. So currently it looks like you are doing check A, then switching midway to check B, then going back to check A again (conceptually).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I agree, pushed that part of code down.
Thanks!

var rules = answerGroup.rules;
rules.forEach(function(rule, ruleIndex) {
var ruleInputs = rule.inputs.x;
ruleInputs.forEach(function(ruleInput) {
if (rule.type === 'IsProperSubsetOf') {
if (ruleInputs.length < 2) {
warningsList.push({
type: WARNING_TYPES.ERROR,
message: (
'In answer group ' + (answerIndex + 1) + ', ' +
'rule ' + (ruleIndex + 1) + ', the "proper subset" ' +
'rule must include at least 2 options.')
});
}
}
});
});
});

areAllChoicesCovered = handledAnswers.every(function(handledAnswer) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure this is correct? You've only considered one type of rule (for the != 1 case). I am not sure you need this actually.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HI! @seanlip,
This rule was considered for all cases, Also The current validation does not consider cases where rule type = "equals", and mininputs>answergroup.length, And when rule type is "contains at least one of", and answergroup.length (number of choices chosen) is 0, both put user into an infinite loop. I feel we can address this in a different issue. And for the All choices covered, I feel it is needed. (tried shifting it up (below the if statement) and running oppia) .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also sorry for late replies again, This will not happen after 16th (sem officially gets over, (ill have end sem exams but most of those are cramming night before)), I will be much more active from 16th.
Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm trying to say is, I'm not sure this check works correctly for the != 1 case. (That's why it was originally only implemented for the == 1 case.)

That's because you're populating handledAnswers only using IsProperSubsetOf and nothing else.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this was another problem, I am not sure how is handled ( I tried checking what is happening in OutcomeObjectFactory,) and how we are checking if all options are handled. The handled choices is used to see if some choice is not handled in the ==1 case, what I am doing is adding all the other choices, but verifying that all are checked (if they have proper subset rule). I think we can solve this in #6418 , by adding other options and validating them. and that way we very each choice is addressed.
Note: yeah all cases are handled but not in a proper way, here if all maxchoices 1 then we verify it here, If all choices are not 1, then that variable is anyway false.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I think I got the question wrong, Yeah the check works for !=1 case, as areAllChoicesCovered will only be true if All choices have been checked by some rule.
Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the logic here. In the maxAllowedCount > 1 case, handledAnswers is initialized here, the entire block from lines 111-147 is skipped (since maxAllowedCount > 1), and then it seems to me like all rule types apart from IsProperSubsetOf are being ignored.

So, why do you say that "areAllChoicesCovered will only be true if all choices have been checked by some rule"? There are other rules besides isProperSubsetOf. I recommend that you do not try to extend that check beyond the == 1 case in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well Yeah I agree, I am confused now, I will remove that for !=1 case,
And this is what I was thinking, when I put it in,
I modified the ItemSelectionValidationService code to this,

if (!areAllChoicesCovered) {     // Now, this will happen if all choices are not covered by =    == 1 case or properSubset rule, I am checking this by not including "selection3"
179 //          if (!defaultOutcome || defaultOutcome.isConfusing(stateName)) {  //this I am guessing che    cks if user has given default feedback (if the user has given oppia something to say, I tried backtracking this, so now Im assuming user has not given a def feedback, So I shuld get this error
180             warningsList.push({
181               type: WARNING_TYPES.ERROR,
182               message: (
183                 'Please add something for Oppia to say in the ' +
184                 '\"All other answers\" response.')
185             });
186 //          }
187         }

and added this check in the Spec file,

 92   it('If everything goes how I except, this case should be handled.', function() {
 93     customizationArguments.maxAllowableSelectionCount.value=1;  //this is the case handled by ===1 ca    se, 
 94     var warnings = validatorService.getAllWarnings(currentState, customizationArguments, IsProperSubs    etValidOption,goodDefaultOutcome);
 95     console.log(warnings);
 96   });
 97 

And checked when options cover everything and when they do not (adding and removing option 3 and checking for warnings) . But yeah this was getting confusing, So I think It is better I drop that, and check for in the ===1 for ProperSubsetRule also case, and make another !=1 case for the proper subset rule.
Ill change it now and commit.

return handledAnswer;
});

if (!areAllChoicesCovered) {
if (!defaultOutcome || defaultOutcome.isConfusing(stateName)) {
Expand Down
Expand Up @@ -68,6 +68,17 @@ describe('ItemSelectionInputValidationService', function() {
false,
null)
];
IsProperSubsetValidOption = [agof.createNew(
[rof.createFromBackendDict({
rule_type: 'IsProperSubsetOf',
inputs: {
x: ['Selection 1']
}
})],
goodDefaultOutcome,
false,
null)
];
}));

it('should be able to perform basic validation', function() {
Expand Down Expand Up @@ -165,4 +176,19 @@ describe('ItemSelectionInputValidationService', function() {
message: 'Please ensure the choices are unique.'
}]);
});

it(
'should expect more that 1 element to be in the rule input, if the ' +
'"proper subset" rule is used.',
function() {
var warnings = validatorService.getAllWarnings(
currentState, customizationArguments, IsProperSubsetValidOption,
goodDefaultOutcome);
expect(warnings).toEqual([{
type: WARNING_TYPES.ERROR,
message: (
'In answer group 1, ' +
'rule 1, the "proper subset" rule must include at least 2 options.')
}]);
});
});
3 changes: 3 additions & 0 deletions extensions/interactions/rule_templates.json
Expand Up @@ -97,6 +97,9 @@
},
"DoesNotContainAtLeastOneOf": {
"description": "omits at least one of {{x|SetOfHtmlString}}"
},
"IsProperSubsetOf": {
"description": "is a proper subset of {{x|SetOfHtmlString}}"
}
},
"LogicProof": {
Expand Down