Browse files

Adding the five help files needed by the 1.4 STABLE

  • Loading branch information...
1 parent 50e752d commit 9b30b693ad2921f1f652b9775775a892bf997b3f rkingdon committed Sep 17, 2004
@@ -0,0 +1,25 @@
+<P ALIGN=CENTER><B>Analysis of Assessments</B></P>
+<P>This analysis looks at the assessments made on the examples as well as those made during the peer assessment phase of the assignment. It attempts to select the better assessments out of this pool of teacher and student assessments. These &quot;good&quot; assesments are then used in the calculation of the final grade.
+<p>This analysis is best done when there are teacher assessments available. These assessments can act as a benchmark against which the student assessments can be judged. The teacher does <b>not</b> need to assess every example and every submission but for the analysis to be meaningful it is better to have more assessments from the teacher than the average number of assessments made by each student. And the more assessments made by the teacher the more confident the teacher can be of the results of the analysis.
+<p>The Analysis is usually done in a number of times, each time changing one or more of the options. The analysis is controlled by the three options which appear on the top of the page.
+<ol><li>The Loading for Teacher Assessments sets the weighting to given the teacher's assessments compared to the students' assessments in the error analysis stage. If the teacher wants their own grading strategy to dominate the way the students grade the submissions then the teacher should be the assessor with the smallest average error in the &quot;Error Table&quot;. If the teacher is not the first one listed than the loading of the teacher's assessments is increased until the teacher has the lowest average error. This then implies that the teacher's assessments are dominate and the students who grade like the teacher are also listed in the top part of the Error Table. The students listed at the bottom part of the table are grading in ways which not match the teacher assessments (nor those of the students in the top of the table). The more assessments that are available from the teacher the more likely that this option will not have to be used to force the teacher to the top of the table. Note that this option does <b>not</b> apply a weighting factor the teacher's assessments when they used in the calculation of final grades. In that calculation the teacher assessments have the same weight as the student assessments. So for example if a student's submission is graded at 41% by the teacher and 45% and 55% by their peers the final grade given to the submission is (41% + 45% + 55%) / 3, that is 47%.</li>
+<li>The Weight for Grading of Assessments is used in the calculation of the Final Grade. A simple formula is used to calculate a student's &quot;Grading Performance&quot;. It is the proportion of &quot;good&quot;assessments that student have done compared to the maximum number of assessments open to them. So, if for example, the assignment asks the students to do 3 assessments of the example submission and 5 peer assessments and the student does 7 assessments and 1 of those is dropped from the analysis (see below), then their grading performance is (7 - 1)/8, that is 75%. The final grade for the assignment a weighted combination of this grading performanace and the grade given to their submission (or best grade if they made more than one submission). The grade for the submission is always given a weight of 1. So setting this option to say, 0.5, means that the two grades are added together in the proportion 0.5:1 or 33% of the grading performance and 66% of the grade of the submission.</li>
+<li>The Percentage of Assessments to drop determines the number of the assessments which are to be excluded when calculating the final grades. This number can be set in one of two ways.
+<ul><li>Given the way the Grading Performance is calculated each student could, if they assessed all the work allocated to them, achieve full marks (for this element) if no assessments are dropped. If the teacher wishes to have a more reason average grade then setting this option to 30% would result in the average Grading Performance of about 70% (again if all students graded all the assessments open to them).</li>
+<li>Alternatively the number of assessments to drop might be set such that the remaining &quot;good&quot; assessments result in the Average Errors being constrained to some reasonable value. These are the percentages given in the fourth column of the Error Table. For example, it may be thought that all the student assessments should (on average) lie within the 20% range. Then the analysis is repeated a number of times adjusting the number of assessments to drop until the figures in this column all lie within a particular limit.
+<p>In addition to the Error Table the analysis lists the grades of all assessments and the final grades given to the students. This table should be inspected to see if the results are reasonable. In particular if many assessments are dropped then some submissions may left unassessed and the student's final grade will be far too small. The analysis does given the number of submissions at the top of page and again just before the Grades Table. These two numbers should be same. If there are one or more unassessed submissions and the teacher does not want to decrease the number of dropped assessments then those submissions should be assessed by the teacher and the analysis repeated. It is important that all submissions are assessed at least once in the final stage of the analysis that is when the final grades are calculated.
+<p>There is a balance between the number of assessments dropped and the overall final grade. The more assessments dropped the lower the final grades are likely to be. However, if poor assessments are <b>not</b> dropped then students may complain about the quality of the assessments which determine the grade for their work. Provided there are enough assessments by the teacher to dominate the analysis without too much forcing, then it would seem reasonable to drop somewhere between 15% and 30% of the assessments.
+<p>Note that this analysis does take a long time as it involves an iterative process. Lengthy delays are to be expected.
@@ -0,0 +1,23 @@
+<P ALIGN=CENTER><B>The Breakdown of the Final Grade</B></P>
+<P>The table on this screen shows your final grade and how it was calculated.
+ The final grade is made up from three possible components.
+<LI>The teacher's grade for your submitted work.
+<LI>The average of the peer grades for your submitted work.
+<LI>A grade for your performance in the peer grading phase. This
+ performance was based by (a) whether your grades were too high or too low
+ when compared with the average grade of the other students (this is called bias),
+ (b) whether your grades follow, again on average, the grades given by
+ the other students (this is called reliability)
+ and (c) on the quality of your comments on the other pieces of work you graded.
+ This was graded by the teacher. These three performance grades were weighted
+ by the factors 1:2:3 respectively to give an overall &quot;grading&quot;
+ grade. In other words the teacher's grading of the comments is given the
+ same weight as the Bias and Reliability factors combined.
+<P>These three components can be weighted as deemed appropriate for the
+ assignment. These weights are shown in the smaller table.
@@ -0,0 +1,54 @@
+<P ALIGN=CENTER><B>The Calculation of the Final Grade</B></P>
+<P>The table on this screen shows how the the final grades for the students
+ are calculated. The final grades are a weighted sum of up to five components.
+<LI>The teacher's grade for their submitted work. This is optional and will be
+ used if the teacher actually assesses the student's work. If the student submits
+ more than one peice of work the &quot;best&quot; grade is used. Here, best
+ means the piece of work with the highest weighted combination of teacher's
+ grade and peer grade...
+<LI>The average of the peer grades for their submitted work. Again if the student
+ submits more than one peice of work the &quot;best&quot; grade is used. The
+ peer grade can optionally
+ include the teacher's grade. This grade would be included if the number of
+ peer gradings is very low or it is thought that the peer gradings are suspect
+ either because of bias (usually on the high side) or for not being reliable.
+ If included the teacher's grade is treated in the same way as a peer grade in
+ the calculation of the average.
+<LI>The student's bias in grading peer work. This is measure of whether the
+ student grades work either too high or too low. It is not an absolute measure
+ as it is based on the difference between the student's grade and the peer
+ averages for each of the submissions they assessed. In general this component
+ should NOT be given a high weighting.
+<LI>The student's reliability in grading peer work. This is a measure on how well
+ a students grades follow the peer average for the peices of work they
+ assessed. The measure discounts the student bias and averages the absolute
+ differences between their grades and the peer average grades. In theory if
+ the students gives high marks for good peices of work and low marks for poor
+ pieces of work their reliability will be high. If it is suspected that the students in
+ general are poor assessors then the teacher's grades should be included into
+ the ppeer averages, this should make the reliability values more meaningful.
+<LI>The average grade given by the teacher for the student's assessments.
+ This includes both the preliminary assessments made by the student on the
+ example pieces of work and any grading the teacher makes on the asessments
+ produced during the peer assessement phase of the assignment. In general this
+ component is probably more important than both the Bias and Reliability
+ components and thus, if available, should be weighted higher.
+<P>These five components can be weighted as deemed appropriate for the
+ assignment. For example the teacher's grade might be weighted strongly
+ if the peer grading part of the assignment is only considered a minor part
+ of the assignment as a whole. Alternatively, if the teacher only grades a few
+ of the submissions these grades can be ignored by giving them a zero weighting.
+ If the assignment is all about the students as judges and the providing of feedback
+ then first two components may be set to zero (or low) and the students'
+ grading abilities will determine the final grades.
+<P>Note that this screen is used iteratively and the final grades are not normally
+ made available to the students until the final phase of the assignment. Once the
+ the teacher is happy with the final grades and their weightings then they can
+ be made available to the students.
@@ -0,0 +1,11 @@
+<P ALIGN=CENTER><B>UnGraded Assessments (Student Submissions)</B></P>
+<P><B>Assessment of Student Submissions</B> These are the peer assessments
+ made by students on eachother's work. In general, these assessments do NOT
+ have to be graded by the teacher. Provided each of the student submissions is
+ assessed about five times, the system can make a reasonable judgement on
+ the inidivual assessment performance of the students. When the number of
+ peer assessments is low then the teacher may want to grade these
+ assessments. Any grades given to the assessments can be taken into account
+ when calculating the final grades for the students.
@@ -0,0 +1,18 @@
+<P ALIGN=CENTER><B>UnGraded Assessments (Teacher Submissions)</B></P>
+<P><B>Assessments of Teacher Submissions</B> These are assessments of
+ the example pieces of work which the assignment may require the students
+ to make before they can submit their own work. These assessments should,
+ in general, be graded by the teacher. The assessments will show whether the
+ student understand the assignment and will provide possibly useful feedback
+ to the teacher on whether any remedial action or fine tuning of the
+ assignment is necessary. Further, if an assessment is graded the teacher's
+ comments are made available to the student. These may provide valuable
+ guidance to the student in the preparation of their own piece of work for
+ the assignment. </P>
+ <P>These assessments do not have to be graded. Leaving a student's
+ assessment of the examples ungraded when NOT stop that student from
+ submitting their own work. It is recommended, however, that all least a
+ sample of the assessments are graded for the reasons mentioned above.</P>

0 comments on commit 9b30b69

Please sign in to comment.