False negatives: repobee reviews check

[connorferster]

Hi there!

I have just started using Repobee in my class and I think it is a life-saver. Thank you for making it and maintaining it!

I did a test-run with my TA where we created an assignment, assigned a review between the two of us (requiring one review), and we each performed a review. We both created "review issues" with the proper issue title (the default, "Peer review"). When I ran the repobee reviews check ... my review showed as complete but his did not. We went back and forth checking to verify that we both did the exact right thing. I adjusted the regex to match anything ".*" but no matter how I ran the check, it kept saying that I had completed the review and he had not.

I then asked him to create two more issues with the same title of "Peer review" and they still did not register as having been completed. I then ran repobee issues list ... with the regex and I could see the reviews that he did but they were not showing up on the check as complete.

Is this something you have encountered before? I am a little bit worried about the false negatives as I have over 80 students in my class. Would be interested to hear your thoughts on what the problem might be!

[slarse]

Hi @connorferster,

Thanks for the kind words!

So, I can't really say off the top of my head what is the matter. Let's try to get to the bottom of it, I have a few questions (but before you answer them read the rest of this message, you may not have to).

1. Which VCS platform are you on (GitHub, GitLab, Gitea)?
2. Can you provde the exact commands that you ran (with options, of course excluding anything sensitive)?
3. When RepoBee assigns reviewers, it creates review teams (groups if on GitLab). Can you verify that your TA is a member of the review team of the repo he is writing up a review for?

Now, we might be able to get to the bottom of why the reviews aren't working for you, but there's an inconvenient fact in that the initial implementation I made of peer reviews is... well, not great. I really wanted to put all of the state on the VCS platform, but in doing so it all became incredibly convoluted and error prone. It turned out to be one of those "seemed like a good idea at the time" kind of things.

There is another option: the RepoBee 4 version of peer review commands are available as a preview feature. They function similarly to the current commands, but the assign command spits out a JSON file that contains the allocations, and all other commands take that as input. It makes for a way more robust review experience. See #868 for details on how to use it. It might even be worthwhile to try that out first, before we spend any significant amount of time trying to debug your current problem.

As the preview commands aren't officially released, any feedback you could provide on them could directly impact how I tweak them for the eventual release. RepoBee 4 was supposed to be released this semester (with a full GUI, as well), but life got in the way, and I don't want to make a major release when semesters have already started.

Sorry for the essay, I just get very excited when people use my stuff :)

[slarse]

Oh, and just to be clear, the reason I recommend trying out the preview commands is that even if they also have some problem, they're a whole lot easier to debug as the review allocations are in a JSON file.

[connorferster]

@slarse

Sorry for the essay, I just get very excited when people use my stuff :)

Lol, I can relate! :)))

Quick answers:

1. GitHub

repobee repos setup -a Workbook_01 --sf test_students.txt
repobee reviews assign -a Workbook_01 --sf test_students.txt --num-reviews 1
repobee reviews check -a Workbook_01 --sf test_students.txt --num-reviews 1 --title-regex '\APeer review\Z'

3. Yes, I can confirm that. We were the only two students in the test_students.txt file.

The JSON approach makes a lot of sense. I did the same thing when I created an adhoc review system that ran out of submissions of assignments to my email inbox. Submissions were auto-filed by Outlook into a sub-folder, running the script would make random pairs of students out of the submitted assignments sub-folder, and then the script would email the reviewing parties the submissions of the other.

Repobee 4, you say? That sounds great. Two questions:

1. So, when I installed repobee this year, it already came with this prototype functionality?
2. How do I set that environment variable?

Looking forward to the next major release!

[slarse]

@connorferster Regarding your answers, everything looks in order ... Can't really say based on that what might be wrong. Let's try the preview commands before we delve any further into debugging.

So, when I installed repobee this year, it already came with this prototype functionality?

Yes, it's been in preview for quite a while, hehe. If you want to ensure that you have the latest version, run repobee manage upgrade, but the preview commands were released in August 2021 so if you installed at any point after that you'll have them.

How do I set that environment variable?

It depends on your enviornment. In bash or any similar shell you'd run export REPOBEE_4_REVIEW_COMMANDS="true". You can verify that they are active by running any RepoBee command and you'll get a warning about using an unstable feature:

[~] $ repobee --help
[WARNING] Activating preview feature REPOBEE_4_REVIEW_COMMANDS

[connorferster]

Thanks very much, @slarse!

I will set the environment variable on the next assignment and see how it goes.

[slarse]

Cool! I'll be very interesting in hearing how it goes.