[Feature:Autograding] Automated output and input generation #3882

weastel · 2019-06-19T12:02:46Z

Please check if the PR fulfills these requirements:

The PR title and message follows our guidelines
Tests for the changes have been added/updated (if possible)
Documentation has been updated/added if relevant

What is the current behavior?

Presently instructor have to provide both output files and input files for grading.

What is the new behavior?

After this PR instructor can provide give grading with both random output and random inputs. They have to add programs in instructor_solution and write commands for input and output generation in config.json.This will generate both inputs and output at time of autograding

Other information?

Install submitty
run BUILD_development.sh script
Run grades for Cpp simple lab Automated input output

…tomated_input_and_output

…tomated_output_generator

emaicus

Nearly ready. A few changes requested.

emaicus · 2019-07-30T14:52:28Z

autograder/autograder/grade_item.py

@@ -604,25 +604,25 @@ def grade_from_zip(my_autograding_zip_file,my_submission_zip_file,which_untruste
            pattern_copy("random_input_to_runner",["*.txt"],random_input_testcase_folder,testcase_folder,tmp_logs)

            # copy compile.out to the current directory


This comment isn't accurate.

changed compile.out to run.out

emaicus · 2019-07-30T14:58:16Z

grading/main_runner.cpp

@@ -25,16 +25,21 @@ int main(int argc, char *argv[]) {
  //If test_case_to_run isn't passed in as a parameter, all testcases are run.
  int test_case_to_run = -1;
  std::string display_variable = "";
-
+  std::string generation_type = "";
+  system("find . -type f -exec ls -sh {} +");


Can we remove this?

emaicus · 2019-07-30T14:58:27Z

grading/main_runner.cpp


  TCLAP::CmdLine cmd("Submitty's main runner program.", ' ', "0.9");
+  system("find . -type f -exec ls -sh {} +");


Same as above.

emaicus · 2019-07-30T15:04:26Z

grading/main_runner.cpp

@@ -98,48 +110,101 @@ int main(int argc, char *argv[]) {
      continue;
    }

-    std::cout << "========================================================" << std::endl;
-    std::cout << "TEST #" << i << std::endl;
+    if ( generation_type == "output" ) {


The clauses of this if statement are nearly duplicates. Can we either break the duplicated portion into the function or else find a way to reduce lines of nearly duplicated code?

Added header function

Can you briefly explain what is the issue?

@emaicus Can you please review it again

…ty/Submitty into automated_input_generator

weastel · 2019-07-31T05:53:21Z

@bmcutler Are test running weird on my branch or they are from master?

…tomated_input_generator

emaicus · 2019-07-31T20:20:48Z

Pushed a commit which:

Pulls output generation commands out of the validator and instead moves them to the be next to commands. This new field is called solution_commands.
Makes output generation play more nicely with graphics and networked gradeables.
1. actions and dispatcher_actions can now be taken against an instructor solution file.
2. solution_commands is inflated to solution_containers at configure time in the same way that commands is inflated to containers. This allows for input to be generated for networked gradeables.
Patches a bug which was causing stdout to be redirected to a STDOUT.txt file even if the instructor specified their own redirect.

This PR should be ready for review.

bmcutler

I may jump in and fix some of these myself
Some of these are notes for myself
I'm not ready to merge just yet.

bmcutler · 2019-08-02T01:03:11Z

autograder/autograder/grade_item.py

+        patterns_work_to_random_output = complete_config_obj["autograding"]["work_to_random_output"]
+        pattern_copy("work_to_random_output", patterns_work_to_random_output, tmp_results, tmp_logs)
+    except:
+        with open(os.path.join(tmp_logs,"overall.txt"),'a') as f:


Why hasn't this been resolved?

bmcutler · 2019-08-02T01:03:50Z

autograder/autograder/grade_item.py

@@ -466,6 +464,60 @@ def grade_from_zip(my_autograding_zip_file,my_submission_zip_file,which_untruste

    os.chdir(tmp_work)

+    #random_input_tmp_work = os.path.join(tmp_work,"random_input")


remove these newly added, commented out lines

autograder/autograder/grade_item.py

bmcutler · 2019-08-02T01:58:23Z

autograder/autograder/grade_item.py

+                                                cwd=random_input_testcase_folder)
+            # remove the compilation program
+            untrusted_grant_rwx_access(which_untrusted, random_input_testcase_folder)
+            os.remove(os.path.join(random_input_testcase_folder,"my_runner.out"))


The generator code & the solution code are still in this folder, right?
Isn't this a risk that the student can read these files?
(solution code especially, should we require that is placed in a separate config folder)

bmcutler · 2019-08-02T01:59:54Z

autograder/autograder/grade_item.py

+
+    with open(os.path.join(tmp_logs,"output_generator_log.txt"), 'w') as logfile:
+        for testcase_num in range(1, len(my_testcases)+1):
+            testcase_folder = os.path.join(tmp_work_random_output, "test{:02}".format(testcase_num))


again, let's add an "if no need to run output generation continue" here

bmcutler · 2019-08-02T02:01:53Z

autograder/autograder/grade_item.py

+
+            # copy test input into testcase folder
+            copy_contents_into(job_id,test_input_path,testcase_folder,tmp_logs)
+            pattern_copy("random_input_to_runner",["*.txt"],random_input_testcase_folder,testcase_folder,tmp_logs)


What if the random input is not a .txt file?
(I know I said we didn't need to worry about this for version 1.0, so maybe we can ignore this, but let's add a comment at least)
This is preventing us from pull generator.py & solution.py into the test case -- but it's sloppy.

bmcutler · 2019-08-02T04:02:54Z

grading/TestCase.cpp

+  assert(containers.size() > 0);
+
+  if (this->CONTAINER_NAME == ""){
+    //TODO add back in if possible.


In this PR or later?

bmcutler · 2019-08-02T04:12:42Z

grading/load_config_json.cpp

    whole_config["autograding"]["work_to_details"].push_back("test*/*_diff.json");
    whole_config["autograding"]["work_to_details"].push_back("**/README.txt");
    whole_config["autograding"]["work_to_details"].push_back("input_*.txt");
    //todo check up on how this works.
    whole_config["autograding"]["work_to_details"].push_back("test*/input_*.txt");
  }

+  if (whole_config["autograding"].find("work_to_random_output") == whole_config["autograding"].end()){


This clause does not make sense to me

bmcutler · 2019-08-02T04:16:18Z

grading/main_runner.cpp

@@ -14,9 +14,85 @@
 // =====================================================================
 // =====================================================================

+std::string addRedirectsToCommand(std::string this_command, std::string which){
+  // Check to see if the instructor is already capturing STDIN
+  if(this_command.find("1>") == std::string::npos){


I think this (and other find's in this function) is fragile to quoted or escaped characters on the command line.
I'm not sure why so much additional code is needed(?)

Was this totally not implemented before? (it may not have been)

…tomated_input_generator

weastel and others added 30 commits May 25, 2019 01:32

output files are now in test_output

2297070

Merge branch 'master' of https://github.com/Submitty/Submitty into au…

7b59f0d

…tomated_input_and_output

solution are included in tmp_work

3e20f93

Merge branch 'master' into automated_input_and_output

a70e466

diff.json is now made

5010751

Merge branch 'master' into automated_input_and_output

1ee749c

change expected_file for imageDiff_doit

cb8b1d9

Fix integration tests to add test_output folder in tmp_work

0049858

Fix json_diff when single parameter is passed

3dbbacd

Fix integration tests for minimal_code_editing

891b5c2

Merge branch 'master' into automated_input_and_output

f0519a5

Merge branch 'master' into automated_input_and_output

0c15391

Fix all tests

dafde2f

Merge branch 'master' into automated_input_and_output

8d5ab77

Merge branch 'master' into automated_input_and_output

e4a9785

Merge branch 'master' into automated_input_and_output

6cb94c9

Merge branch 'master' into automated_input_and_output

85c4b38

Change solution directory to instructor_solution directory

ad37dbc

genrating ouput folder in generated_output

3e34df2

Merge branch 'master' of https://github.com/Submitty/Submitty into au…

e162ac5

…tomated_output_generator

Add new example for automated output as cpp_hidden_tests

e6c550e

updating functions for instructor uses

3e86881

Merge branch 'master' into automated_output_generator

fdc88b0

View generated output in details

2b0d0c2

Merge branch 'master' into automated_output_generator

b89622d

Automated output example for python in more autograding example

90e4de4

Merge branch 'master' into automated_output_generator

80ce45f

Change generated_output to random_output

a89a778

View expected file for random output

fac1381

Merge branch 'master' into automated_output_generator

532136c

remove additional lines of code

9b2a2ff

weastel requested review from bmcutler and emaicus July 27, 2019 09:40

weastel and others added 2 commits July 28, 2019 03:41

Merge branch 'master' into automated_input_generator

068e62d

Merge branch 'master' into automated_input_generator

de6ad20

emaicus requested changes Jul 30, 2019

View reviewed changes

add header function and change comments

35ee6f3

weastel requested a review from emaicus July 30, 2019 16:30

weastel and others added 4 commits July 30, 2019 22:05

update comments

1021580

Merge branch 'master' into automated_input_generator

0ac6399

Remove header and add executeSetOfCommands

3dd2867

Merge branch 'automated_input_generator' of https://github.com/Submit…

a016f78

…ty/Submitty into automated_input_generator

bmcutler changed the title ~~[SYSADMIN ACTION][Feature] Automated output and input generation~~ [Feature:Autograding] Automated output and input generation Jul 31, 2019

emaicus added 3 commits July 31, 2019 12:02

Move output generation out of the validator

8a1329c

Fixed merge conflicts

cc20733

Merge branch 'master' of https://github.com/Submitty/Submitty into au…

ef0dcc8

…tomated_input_generator

bmcutler reviewed Aug 2, 2019

View reviewed changes

bmcutler and others added 8 commits August 2, 2019 00:21

Merge branch 'master' into automated_input_generator

6d9ba20

update

ea46b7e

update and add comments

53eb35b

Remove 'autograding' entry for work_to_random_output

31dd3ba

Remove work_to_random_output

c17d690

Merge branch 'master' of https://github.com/Submitty/Submitty into au…

4f14cd0

…tomated_input_generator

Fixes

aa18c5e

Merge branch 'master' of https://github.com/Submitty/Submitty into au…

199ad1c

…tomated_input_generator

bmcutler merged commit 7e7adf0 into master Aug 2, 2019

bmcutler deleted the automated_input_generator branch August 2, 2019 16:37

weastel mentioned this pull request Aug 3, 2019

[BUGFIX:Tests] Validate solution_containers in complete_config json #4280

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature:Autograding] Automated output and input generation #3882

[Feature:Autograding] Automated output and input generation #3882

weastel commented Jun 19, 2019 •

edited by bmcutler

emaicus left a comment

emaicus Jul 30, 2019

weastel Jul 30, 2019

emaicus Jul 30, 2019

emaicus Jul 30, 2019

emaicus Jul 30, 2019

weastel Jul 30, 2019

weastel Jul 30, 2019

weastel Jul 31, 2019

weastel commented Jul 31, 2019

emaicus commented Jul 31, 2019

bmcutler left a comment

bmcutler Aug 2, 2019

bmcutler Aug 2, 2019

bmcutler Aug 2, 2019

bmcutler Aug 2, 2019

bmcutler Aug 2, 2019

bmcutler Aug 2, 2019

bmcutler Aug 2, 2019

bmcutler Aug 2, 2019

bmcutler Aug 2, 2019

		@@ -604,25 +604,25 @@ def grade_from_zip(my_autograding_zip_file,my_submission_zip_file,which_untruste
		pattern_copy("random_input_to_runner",["*.txt"],random_input_testcase_folder,testcase_folder,tmp_logs)

		# copy compile.out to the current directory


		TCLAP::CmdLine cmd("Submitty's main runner program.", ' ', "0.9");
		system("find . -type f -exec ls -sh {} +");

		@@ -466,6 +464,60 @@ def grade_from_zip(my_autograding_zip_file,my_submission_zip_file,which_untruste

		os.chdir(tmp_work)

		#random_input_tmp_work = os.path.join(tmp_work,"random_input")

[Feature:Autograding] Automated output and input generation #3882

[Feature:Autograding] Automated output and input generation #3882

Conversation

weastel commented Jun 19, 2019 • edited by bmcutler

Please check if the PR fulfills these requirements:

What is the current behavior?

What is the new behavior?

Other information?

emaicus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weastel commented Jul 31, 2019

emaicus commented Jul 31, 2019

bmcutler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weastel commented Jun 19, 2019 •

edited by bmcutler