What is this?

In this repository are two small programs that implement the first two stages of the Delphi evaluation method¹.

Usually, the first two stages are paper-based. Each student would receive a sheet of paper, note three negative comments on top of the paper and three positive comments in the bottom of it. After everybody is done commenting all sheets are rotated clock-wise through the classroom. When a student agrees on a comment from one of her class-mates she indicates that by setting a tick next to it. Once each student receive their own sheet of paper again phase one is stopped, you would collect all papers, count the most prominent issues (negative comments) and discuss the four/five most prominent issues in class with the students. I focus on the negative ones as they allow you to improve in future.

Now with COVID-19 and online learning, the above process can obviously not be executed. The programs (phase_[one|two]_server.go) in this repository try to implement an online version of it. The programs are as simple as I could make them. First, to get them implemented quickly and second to fit my needs.

Note, for this tool, I decided to allow for only up to three positive and three negative comments per student.

How to run an evaluation?

• To run an evaluation you need two executables for your operating system (phase_[one|two]_server for Linux, phase_[one|two]_server.exe for Windows, and phase_[one|two]_server_mac for MacOS respectively), which you find in the build directory.

• You run the evaluation in two phases.

• Phase 1:

  • First, you start the server for phase one, which serves a web-page to the students on which they can comment on the course that you want to evaluate. On Linux you may want to start it like the following:

  ./phase_one_server --addr=0.0.0.0 --port=8888 >> log/phase_one.log 2>&1 &

  • Now, you can point the students to the address of the machine together with the port on which this application is running. In case you want to run the the evaluation directly from your computer, you might want to have a look at tools like ngrok.
  • The page served to the students looks like in the following:

  

  • This first phase of evaluation produces a text file responses.txt (actually it should be a CSV file but for now the program does not guarantee that it well-formed) directly next to where the binary phase_one_server is located. It contains all positive and negative comments from all students.
  • Once the students are done providing their comments you can stop the first phase of the evaluation, for example, on Linux via:

  $ ps aux | grep phase
  root     13367  0.0  0.1  14856  1092 pts/1    S+   08:53   0:00 grep --color=auto phase
  root     30696  0.0  0.4 798332  4716 ?        Sl   Apr29   0:00 ./phase_one_server --addr=0.0.0.0 --port=8888
  $ kill -TERM 30696

  • The contents of the generated responses.txt file look similar to the following (example comments):

  2020-04-29T16:10:09Z,<source_ip>,neg_c,"I did not like it at all."
  2020-04-29T16:10:09Z,<source_ip>,pos_a,"Great freedom to develop our application as we wanted"
  2020-04-29T16:10:09Z,<source_ip>,pos_b,"The simulator was fun because it felt like real users"
  2020-04-29T16:10:09Z,<source_ip>,pos_c,"The use of industry relevant tools was great"
  2020-04-29T16:10:09Z,<source_ip>,neg_a,"Course was quite hectic at the start"
  2020-04-29T16:10:09Z,<source_ip>,neg_b,"The git lecture was meaningless"
  2020-04-29T16:15:18Z,<other_source_ip>,pos_a,"Lecturers and TAs are really good at presenting relavant stuff, the contents are great!"
  2020-04-29T16:15:18Z,<other_source_ip>,pos_b,"Guest lecture was great"
  2020-04-29T16:15:18Z,<other_source_ip>,pos_c,""
  2020-04-29T16:15:18Z,<other_source_ip>,neg_a,"You should maybe explain contents better."
  2020-04-29T16:15:18Z,<other_source_ip>,neg_b,""
  2020-04-29T16:15:18Z,<other_source_ip>,neg_c,""
  

• Phase 2:

  • Now, you start the server for phase two, which serves a web-page to the students on which they can indicate on which comments they agree. On Linux you may want to start it like the following:

  ./phase_two_server --addr=0.0.0.0 --port=8888 >> log/phase_two.log 2>&1 &

  • Note, this binary depends on the file responses.txt that was generated in the previous phase. That is, start the binary for the second phase next to where you ran the first one (same directory).
  • The page served to the students looks like in the following:
  • There, they can set check-marks next to each comment on which they agree.
  • Once the students are done providing their ratings you can stop the second phase of the evaluation, similar to above.
  • This second phase of evaluation produces a text file rated_responses.txt, which contains a line for each comment that was checked respectively.
  • The contents of the generated rated_responses.txt file look similar to the following (example comments; note one line per given check mark ):

  neg_45,"This is the 7.5 ECTS course i have spent the most time on since i started at itu. "
  neg_51,"Some parts has already been covered by other courses"
  neg_45,"This is the 7.5 ECTS course i have spent the most time on since i started at itu. "
  neg_51,"Some parts has already been covered by other courses"
  pos_30,"Most excercises made sense on the weeks which they were introduced. only a very few odd cases."
  neg_45,"This is the 7.5 ECTS course i have spent the most time on since i started at itu. "
  pos_30,"Most excercises made sense."
  

• To quickly find the most frequent negative issues to discuss in the plenum, I ran:

sort rated_responses.txt | uniq -c | sort -r | grep "neg_"

Which creates something similar to the following, where the numbers in the first column state how many times a certain negative comment was 'up-voted':

26 neg_5,"Long lectures, no time for group work"
24 neg_6,"Do not use Talk, use LearnIT instead"
23 neg_15,"Some exercises are too large to get up and running within a week."
21 neg_53,"Not a fan of 4 hour sessions"

---

Technical Notes

• I only tested the executables for Linux on Ubuntu 18.04 so far.
• Be aware of that all text is transfered in plain-text for the moment (no HTTPS and in the first phase responses are send to the server encoded in the URL). That is, tell the students to not send sensitive data via the forms (which they likely would not do in the public physical paper-based setup either :))
• Currently, the system is dumb in the sense that it does not prevent students from submitting comments or ratings multiple times. Similar to the paper-based process you have to trust on that the students behave nicely.

Building it yourself.

• Checkout this repository
• cd delphi_evaluator
• make

Requirements

• You need the Go programming language installed and setup

1:

• http://www.marinos.dk/eva/
• http://www.marinos.dk/eva/Skriftlig-B-Delphi-metoden%20(med%20et%20papir%20til%20hele%20klassen).doc
• https://educate.au.dk/praksiseksempler/undervisningsevaluering/lynevaluering/
• http://www.irenelarsen.dk/Evaluering/delphi-metoden.pdf