This repository has been archived by the owner. It is now read-only.

Do you use automated testing/grading for assignments? #3

Open
afeld opened this Issue Jun 18, 2014 · 19 comments

Comments

Projects
None yet
@afeld
Contributor

afeld commented Jun 18, 2014

Do you use any autograder or automated testing with with students? Do they write the tests, or do you? Are they able to see the tests or the output? What software do you use to run them?

@afeld afeld added the open ended label Jun 18, 2014

@hpssjellis

This comment has been minimized.

Show comment
Hide comment
@hpssjellis

hpssjellis Jun 19, 2014

Generally I don't trust the students to do any real online testing. They creatively cheat even when I am watching. I have however spent a ton of time making an online Physics 11 pratice exam at http://142.33.222.23/ellis/cpTests/ph11Exam.html which randomly sets questions and gives answers at the bottom of the page.

Generally I don't trust the students to do any real online testing. They creatively cheat even when I am watching. I have however spent a ton of time making an online Physics 11 pratice exam at http://142.33.222.23/ellis/cpTests/ph11Exam.html which randomly sets questions and gives answers at the bottom of the page.

@afeld

This comment has been minimized.

Show comment
Hide comment
@afeld

afeld Jun 20, 2014

Contributor

Sorry, I could have been more clear: I meant automated tests for their code (of the JUnit, RSpec, QUnit, etc. variety).

Contributor

afeld commented Jun 20, 2014

Sorry, I could have been more clear: I meant automated tests for their code (of the JUnit, RSpec, QUnit, etc. variety).

@hpssjellis

This comment has been minimized.

Show comment
Hide comment
@hpssjellis

hpssjellis Jun 20, 2014

Oops. Can you delete my post if you think it is not relevant. I don't use
browser testers since I compile a working chrome javascript page into an
App using Phonegap.com. If it doesn't work then something is wrong.

On Thu, Jun 19, 2014 at 8:43 PM, Aidan Feldman notifications@github.com
wrote:

Sorry, I could have been more clear: I meant automated tests for their
code (of the JUnit, RSpec, QUnit, etc. variety).


Reply to this email directly or view it on GitHub
#3 (comment).

Oops. Can you delete my post if you think it is not relevant. I don't use
browser testers since I compile a working chrome javascript page into an
App using Phonegap.com. If it doesn't work then something is wrong.

On Thu, Jun 19, 2014 at 8:43 PM, Aidan Feldman notifications@github.com
wrote:

Sorry, I could have been more clear: I meant automated tests for their
code (of the JUnit, RSpec, QUnit, etc. variety).


Reply to this email directly or view it on GitHub
#3 (comment).

@blake41

This comment has been minimized.

Show comment
Hide comment
@blake41

blake41 Jun 21, 2014

We have automated tests for almost all our assignments. In some cases as the course progresses they write their own tests. I'd like them to write more of their own tests but the automated testing allows them to get immediate feedback.

blake41 commented Jun 21, 2014

We have automated tests for almost all our assignments. In some cases as the course progresses they write their own tests. I'd like them to write more of their own tests but the automated testing allows them to get immediate feedback.

@georgeyiu

This comment has been minimized.

Show comment
Hide comment
@georgeyiu

georgeyiu Jul 14, 2014

We use JUnit for much of our Java testing. We provide a subset of our autograder suite to the students at the beginning of the project and require that they extend that suite and write more tests. If the project is well spec'ed, you can incorporate some of the better student tests into the autograder suite for future terms in the course. Another great method of encouraging testing is writing a version of the master solution code that includes several bugs and requiring students to write tests to break them (blindly, without telling them the bugs of course).

We use JUnit for much of our Java testing. We provide a subset of our autograder suite to the students at the beginning of the project and require that they extend that suite and write more tests. If the project is well spec'ed, you can incorporate some of the better student tests into the autograder suite for future terms in the course. Another great method of encouraging testing is writing a version of the master solution code that includes several bugs and requiring students to write tests to break them (blindly, without telling them the bugs of course).

@alexjacoby

This comment has been minimized.

Show comment
Hide comment
@alexjacoby

alexjacoby Aug 20, 2014

I'd love to hear if/how anyone is using webhooks to automate testing of student submissions. I've skimmed the documentation for the automated testing services but I'd love to hear what people think works best for this situation.

I'd love to hear if/how anyone is using webhooks to automate testing of student submissions. I've skimmed the documentation for the automated testing services but I'd love to hear what people think works best for this situation.

@delucas

This comment has been minimized.

Show comment
Hide comment
@delucas

delucas Aug 24, 2014

Hey @alexjacoby! I'm using a custom tool (a daemon that pulls homework from a queue, which is populated when my students make a push into their GitHub repositories with the answer).
It pulls the repo and then runs a series of automated tests (JUnit, PMD, Checkstyle and Cobertura). Then it takes the output and converts it into a unified score.
What do you think to use? Travis-CI seems good if you have a nice custom script that gets a score out of the test results.

delucas commented Aug 24, 2014

Hey @alexjacoby! I'm using a custom tool (a daemon that pulls homework from a queue, which is populated when my students make a push into their GitHub repositories with the answer).
It pulls the repo and then runs a series of automated tests (JUnit, PMD, Checkstyle and Cobertura). Then it takes the output and converts it into a unified score.
What do you think to use? Travis-CI seems good if you have a nice custom script that gets a score out of the test results.

@afeld

This comment has been minimized.

Show comment
Hide comment
@afeld

afeld Aug 24, 2014

Contributor

Should have mentioned before, but there's a section of the classroom guide about automated testing, as well as a description of how I'm using Travis etc. in one of my classes.

Contributor

afeld commented Aug 24, 2014

Should have mentioned before, but there's a section of the classroom guide about automated testing, as well as a description of how I'm using Travis etc. in one of my classes.

@neurodude

This comment has been minimized.

Show comment
Hide comment
@neurodude

neurodude Oct 27, 2014

Any Idea how to integrate travis-ci with unit testing for the student submissions. I have been trying to work out how to run the unit tests without giving them the unit tests to begin with. Any thoughts?

Any Idea how to integrate travis-ci with unit testing for the student submissions. I have been trying to work out how to run the unit tests without giving them the unit tests to begin with. Any thoughts?

@afeld

This comment has been minimized.

Show comment
Hide comment
@afeld

afeld Oct 27, 2014

Contributor

I don't know of any way to do that through Travis – would probably need to set up something custom if hiding the tests is important to you.

Contributor

afeld commented Oct 27, 2014

I don't know of any way to do that through Travis – would probably need to set up something custom if hiding the tests is important to you.

@neurodude

This comment has been minimized.

Show comment
Hide comment
@neurodude

neurodude Oct 28, 2014

In the classroom guide it mentions auto grading. In that context it says:

Unit tests

Tests can be written by you, and/or by the students
Hidden from the student (a.k.a. an "autograder") or not

Any idea what they were referring to or how to set that up?

In the classroom guide it mentions auto grading. In that context it says:

Unit tests

Tests can be written by you, and/or by the students
Hidden from the student (a.k.a. an "autograder") or not

Any idea what they were referring to or how to set that up?

@afeld

This comment has been minimized.

Show comment
Hide comment
@afeld

afeld Oct 28, 2014

Contributor

Right. The basic idea is that you need (private) scripts run when the code is submitted, which can be triggered via web hooks. https://travis-ci.org/ can't be used because there's no way to upload the hidden tests, but any of those other options (hosted by you, at least) should be able to support it. I don't know of any specific guides for setting something up w/ GitHub, but searching for "set up autograder" had a bunch of other examples.

@armandofox Saw your blog post about autograding at scale... what did you end up doing?

Contributor

afeld commented Oct 28, 2014

Right. The basic idea is that you need (private) scripts run when the code is submitted, which can be triggered via web hooks. https://travis-ci.org/ can't be used because there's no way to upload the hidden tests, but any of those other options (hosted by you, at least) should be able to support it. I don't know of any specific guides for setting something up w/ GitHub, but searching for "set up autograder" had a bunch of other examples.

@armandofox Saw your blog post about autograding at scale... what did you end up doing?

@neurodude

This comment has been minimized.

Show comment
Hide comment
@neurodude

neurodude Oct 28, 2014

How would I do it if I was ok with them seeing the tests? I have now a gradle build building and testing junits. Do I have to post the whole file structure? Can I just put the tests somewhere and have them run?

Also, It appears that google is different for you and me. I don't get other examples when I search "set up autograder"?

How would I do it if I was ok with them seeing the tests? I have now a gradle build building and testing junits. Do I have to post the whole file structure? Can I just put the tests somewhere and have them run?

Also, It appears that google is different for you and me. I don't get other examples when I search "set up autograder"?

@efthym

This comment has been minimized.

Show comment
Hide comment
@efthym

efthym Jan 20, 2015

Hi All,

I'm preparing to use github for a University course following the sandbox approach.
I got myself fairly familiar with teachers_pet and it worked fine for me so far.

Now, I would like to add the capability of running Travis-CI when students push their work to their (private) repos so that an automatic script in Ruby checks their work and they get to know if their submission meets at least basic requirements.
I'm happy to give the script, which contains the expected results, to the students, but I plan to run other tests and visually inspect their work.

I know teachers_pet push_files copies everything from a local repo to multiple student repositories so I can get .travis.yml files into their repositories.
However, I'd like them to be able to see the passing/failing "badge" in their own README.md files, so they can instantly know if their work is OK.
Unfortunately, these badges need URLs which contain unique tokens for private repositories...
Is there an easy way to customise each student's README.md file so it contains their own Travis-CI token? (assuming there's a way of getting it automatically).

Do you have any other suggestions for showing build status/automarker result in a private student repo?

Thanks,

Aris

efthym commented Jan 20, 2015

Hi All,

I'm preparing to use github for a University course following the sandbox approach.
I got myself fairly familiar with teachers_pet and it worked fine for me so far.

Now, I would like to add the capability of running Travis-CI when students push their work to their (private) repos so that an automatic script in Ruby checks their work and they get to know if their submission meets at least basic requirements.
I'm happy to give the script, which contains the expected results, to the students, but I plan to run other tests and visually inspect their work.

I know teachers_pet push_files copies everything from a local repo to multiple student repositories so I can get .travis.yml files into their repositories.
However, I'd like them to be able to see the passing/failing "badge" in their own README.md files, so they can instantly know if their work is OK.
Unfortunately, these badges need URLs which contain unique tokens for private repositories...
Is there an easy way to customise each student's README.md file so it contains their own Travis-CI token? (assuming there's a way of getting it automatically).

Do you have any other suggestions for showing build status/automarker result in a private student repo?

Thanks,

Aris

@efthym efthym referenced this issue in education/teachers_pet Jan 25, 2015

Closed

travis auto testing and delete repository command #102

@motoxman36

This comment has been minimized.

Show comment
Hide comment
@laprej

This comment has been minimized.

Show comment
Hide comment
@laprej

laprej Jan 26, 2016

So if I want my students to use Travis while using the sandbox approach, I should have them manually enable it on each repo, correct?

laprej commented Jan 26, 2016

So if I want my students to use Travis while using the sandbox approach, I should have them manually enable it on each repo, correct?

@Omarasifshaikh

This comment has been minimized.

Show comment
Hide comment
@Omarasifshaikh

Omarasifshaikh Feb 17, 2016

Hi Aris @efthym
Were you able to figure out your problem? As of now, Travis CI has granted educational access to my class repository, so students can head over to travis-ci.com and run tests on the private repository.

@laprej Yes I believe students will have to log in into Travis-ci.com and enable the build for each repository.

Hi Aris @efthym
Were you able to figure out your problem? As of now, Travis CI has granted educational access to my class repository, so students can head over to travis-ci.com and run tests on the private repository.

@laprej Yes I believe students will have to log in into Travis-ci.com and enable the build for each repository.

@Omarasifshaikh

This comment has been minimized.

Show comment
Hide comment
@Omarasifshaikh

Omarasifshaikh Feb 17, 2016

As of now, I am doing the following(for my advanced C++ class):

  • Use GitHub classroom to create individual private repositories
  • These repositories have CMake and Google's Test library incorporated, so the students can use CLion to test locally.
  • The Students can also head to Travis-ci.com and enable cloud testing for themselves.
  • My TA's can see the list of all the private repositories and hence see if they pass or fail.
  • Here is an example of a C++ factorial program that uses Google test and Travis CI:

https://github.com/csc340-03-spring-2016/factorials-cpp-gtest-travisci.git

Still to figure out:

  • Grade within Travis to determine score based on number of C++ tests passed
  • Use Travis to clone and diff the test folder to determine plagiarism(changes to the test folder)
  • Hide certain tests.

As of now, I am doing the following(for my advanced C++ class):

  • Use GitHub classroom to create individual private repositories
  • These repositories have CMake and Google's Test library incorporated, so the students can use CLion to test locally.
  • The Students can also head to Travis-ci.com and enable cloud testing for themselves.
  • My TA's can see the list of all the private repositories and hence see if they pass or fail.
  • Here is an example of a C++ factorial program that uses Google test and Travis CI:

https://github.com/csc340-03-spring-2016/factorials-cpp-gtest-travisci.git

Still to figure out:

  • Grade within Travis to determine score based on number of C++ tests passed
  • Use Travis to clone and diff the test folder to determine plagiarism(changes to the test folder)
  • Hide certain tests.
@efthym

This comment has been minimized.

Show comment
Hide comment
@efthym

efthym Feb 17, 2016

Hi Omar @Omarasifshaikh
No I haven't done anything on this front yet.
I think @laprej is right, though I'm not sure if it requires that students get the student pack, so they can have private repos of their own on GitHub.

efthym commented Feb 17, 2016

Hi Omar @Omarasifshaikh
No I haven't done anything on this front yet.
I think @laprej is right, though I'm not sure if it requires that students get the student pack, so they can have private repos of their own on GitHub.

@troeger troeger referenced this issue in troeger/opensubmit Mar 21, 2016

Open

Support automated code grading #1

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.