Automated Assignment Grading System with AI Model Evaluation

Objective:

• Develop an automated assignment grading system to reduce manual grading fatigue and improve efficiency.
• Evaluate the reliability of various AI models in grading assignments as part of a research initiative.

Project Steps:

1. Project Setup:

   • Establish a project timeline and milestones.
   • Identify the necessary technological stack and resources, including RAG, LangChain, and LLM models.

2. Data Input Module:

   • Develop an input module that accepts a zip file containing text-based assignments (Word, PDF, Python notebooks).
   • Design a mechanism to extract and process text from these formats for grading.

3. Grading Criteria Definition:

   • Implement a system allowing graders to define specific grading criteria and processes for each assignment type.
   • Ensure the system supports diverse grading metrics and workflows tailored to different assignments.

4. AI Grading System:

   • Integrate AI automation flows to grade assignments based on the predefined criteria.
   • Employ multiple AI models and compare their grading outputs to ensure reliability and consistency.

5. Grading Consistency and Verification:

   • Define a protocol for grading each assignment multiple times to verify consistency.
   • Develop an algorithm to compare grading outcomes across iterations, flagging significant discrepancies for review.

6. Flagging System:

   • Create a web interface to display flagged assignments, highlighting variations in grading.
   • Allow graders to review and adjust grades for flagged assignments, ensuring accuracy.

7. Result Compilation and Reporting:

   • Generate comprehensive reports detailing each assignment's grade, confidence levels, and variation across grading iterations.
   • Include constructive feedback based on grading metrics, offering insights into areas of improvement for students.

8. Testing and Iteration:

   • Conduct thorough testing of the system with a variety of assignment samples.
   • Gather feedback from potential users and refine the system based on this input.

9. Future Expansion:

   • Plan for the incorporation of more complex assignment types, including those with embedded images or code, in subsequent phases of the project.

10. Documentation and Dissemination:

    • Document the development process, system architecture, and user instructions.
    • Prepare a paper or presentation to share the research findings on AI model reliability in grading.

Expected Outcomes:

• An operational automated grading system that enhances grading efficiency and consistency.
• Insights into the effectiveness and reliability of different AI models in the educational grading context.

Detailed Documentation

• Check out the project Wiki and list of issues for more context.
• Check out CONTRIBUTING.md for contribution guidelines