Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LLM Autograding manual graded questions #9887

Merged
merged 53 commits into from
Jun 21, 2024

Conversation

Victorsss-Orz
Copy link
Contributor

This is a part of the SIIP grant for Automatic Short Answer Grading using Large Language Models. The first PR works on the real basic functions, and more features will be added later.
We added a grade button in the manual grading assessment question page that utilizes large language models to autograde all problems, and update the scores and feedbacks.
image
image
image
image

Copy link

codecov bot commented May 24, 2024

Codecov Report

Attention: Patch coverage is 18.48341% with 172 lines in your changes missing coverage. Please review.

Project coverage is 69.01%. Comparing base (dfa97d9) to head (83e9b51).

Current head 83e9b51 differs from pull request most recent head 100d658

Please upload reports for the commit 100d658 to get more accurate results.

Files Patch % Lines
apps/prairielearn/src/lib/bot-grading.ts 16.12% 156 Missing ⚠️
...alGrading/assessmentQuestion/assessmentQuestion.ts 15.78% 16 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9887      +/-   ##
==========================================
- Coverage   69.23%   69.01%   -0.23%     
==========================================
  Files         484      486       +2     
  Lines       76262    76380     +118     
  Branches     6447     6439       -8     
==========================================
- Hits        52799    52710      -89     
- Misses      23012    23219     +207     
  Partials      451      451              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@jonatanschroeder jonatanschroeder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is by no means a complete or formal review, I'm only providing some initial thoughts.

@SethPoulsen SethPoulsen self-requested a review May 28, 2024 16:07
Copy link
Collaborator

@SethPoulsen SethPoulsen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been working closely with Victor throughout this PR, so most of the comments and changes, I've just shared with him directly offline.

There is obviously a lot more work to do in order to make these features work well, but I think this represents a good MVP for something that works minimally enough so that we should merge it and make additional changes incrementally.

@SethPoulsen
Copy link
Collaborator

@Victorsss-Orz it looks like you have conflicts with the yarn.lock file. One way to fix this would be to re-check out the version of yarn.lock from the master branch, and then re-run your yarn add openai command. There might be a faster way than that, I don't know though.

@Victorsss-Orz
Copy link
Contributor Author

@Victorsss-Orz it looks like you have conflicts with the yarn.lock file. One way to fix this would be to re-check out the version of yarn.lock from the master branch, and then re-run your yarn add openai command. There might be a faster way than that, I don't know though.

Done

apps/prairielearn/src/lib/bot-grading.ts Outdated Show resolved Hide resolved
apps/prairielearn/src/lib/bot-grading.ts Outdated Show resolved Hide resolved
apps/prairielearn/src/lib/bot-grading.ts Outdated Show resolved Hide resolved
apps/prairielearn/src/lib/bot-grading.ts Outdated Show resolved Hide resolved
apps/prairielearn/src/lib/bot-grading.ts Outdated Show resolved Hide resolved
apps/prairielearn/src/lib/bot-grading.ts Outdated Show resolved Hide resolved
apps/prairielearn/src/lib/bot-grading.ts Outdated Show resolved Hide resolved
apps/prairielearn/src/lib/bot-grading.ts Show resolved Hide resolved
apps/prairielearn/src/lib/bot-grading.ts Outdated Show resolved Hide resolved
apps/prairielearn/src/lib/bot-grading.ts Show resolved Hide resolved
Copy link
Contributor

@nwalters512 nwalters512 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more comments, then this should be good to merge.

@Victorsss-Orz
Copy link
Contributor Author

Looking forward to getting this merged

@nwalters512 nwalters512 added this pull request to the merge queue Jun 21, 2024
Merged via the queue into PrairieLearn:master with commit 780ce90 Jun 21, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants