Full Judge Solution for Testing the Judge Locally without Calling the API

[Mahboob-A]

Clone this branch if you just want to test how the Judge is working locally without calling the API and passing any data.

Few demo codes are attached in dummy_questions.py to test the Judge locally without calling the API.
You can tweak the questions as per your need.

How to Run:

A.
i. Clone the check-dnd-local branch.

ii. Create a virtual env and activate it.

iii. cd src

iv. Create the C++ Image. (If you already have not build the same image)

a. cd /docker/lang/cpp/

b. Run: docker build -t algocode/cpp:v1 .
Make sure to put a "dot" at the end of the line. This will create the necessary image we need to compile the C++ code.

iv. Create the volume that will be mapped with the Judge Container and Sibling Container.
Run: docker volume create user_code_files

iv. CD to src and Run: make docker-up to run the Docker Compose file (runs the main Judge Container).

B.
i. Run docker ps

And you will see a container is running with the name src-judge-1

ii. Copy the first 5 digits of the container ID

iii. Run docker exec -it first_5_digit_container_id /bin/bash
You are now into the main Judge Container.

Now you need to go to the views.py file location

v. cd /core_apps/judge_engine You are now inside the Judge Engine APP.

vi. Run python views.py

You will see a output something like below depending upon the dummy_question is passed.

For AC:

Entrypoint of User's Unique Data Dir Judge Container Mount Point:  /app/user-files/user_codes/cpp/cf553272-611d-4207-8463-e91c4e179a04
data: 
 {'result': {'result': 'All testcases passed', 'verdict': 'Accepted'}, 'status_code': 0, 'status': 'Successful', 'error_message': None, 'success_message': 'code compiled and run successfully.'}

For WA:

Entrypoint of User's Unique Data Dir Judge Container Mount Point:  /app/user-files/user_codes/cpp/ad47e0ba-ae16-4389-9d9c-8499c42fa1d9
data: 
{'result': {'result': 'Wrong Answer at test case 10', 'Output': 'Hello World: 10', 'Expected': 'Hello World: 0', 'verdict': 'Wrong Answer'}, 'status_code': 0, 'status': 'Successful', 'error_message': None, 'success_message': 'code compiled and run successfully.'}

vii. CD to src and Run make docker-down to stop the container.

Note: You can change the question in the test_inside_cont_exe function's user_codes parameter.

[sweep-ai]

Sweep: PR Review

Authors of pull request: @Mahboob-A

This pull request aimed to enhance the Docker-based development environment and refactor the code execution engine for better modularity and maintainability.

A Makefile was added to simplify Docker Compose commands, introducing dup and ddown targets for starting and stopping the development environment. The dev.yml Docker Compose file was created to define the services, networks, and volumes required for the development setup, including a judge service.

The containers.py file was updated to change the Docker image used for running containers and to persist containers by commenting out the stop and remove commands. The exec_engine.py and file_data_processor.py files were refactored to handle file paths and volume mounts more effectively, introducing variables like judge_volume_mount for better path management.

The views.py file saw significant changes, with the existing API views being commented out and replaced by a local testing function test_inside_cont_exe to facilitate local testing of the code execution engine.

New Dockerfiles were added for both the Django application and the C++ execution environment, with changes to the directory structure and volume mounts to ensure consistency across the development environment.

Overall, these changes aimed to streamline the development workflow, improve code modularity, and enhance the local testing capabilities of the code execution engine.

---

Sweep Found These Issues

src/core_apps/judge_engine/containers.py

• Changing the import source of TimeLimitExceedException to exceptions may cause runtime errors if the new module does not define the exception class correctly.

online-judge/src/core_apps/judge_engine/containers.py

Lines 12 to 13 in 960a0f3

	# from core_apps.judge_engine.exceptions import TimeLimitExceedException
	from exceptions import TimeLimitExceedException


View Diff
• Changing the import source for TimeLimitExceedException to exceptions may cause runtime errors if the new module does not define the exception class correctly.

online-judge/src/core_apps/judge_engine/containers.py

Lines 12 to 13 in 960a0f3

	# from core_apps.judge_engine.exceptions import TimeLimitExceedException
	from exceptions import TimeLimitExceedException


View Diff

src/core_apps/judge_engine/exec_engine.py

• The introduction of judge_volume_mount and the modification of parent_dir could lead to incorrect file paths if judge_volume_mount is not correctly set or if the split operation fails.

online-judge/src/core_apps/judge_engine/exec_engine.py

Lines 113 to 123 in 960a0f3

	judge_volume_mount,
	) = result[1:]
	# code_filepath: base-dir/user_codes/lang/uuid/main.cpp
	# parent_dir: base-dir/user_codes/lang/uuid
	# /app/user-files/user_codes/cpp/uuid
	parent_dir = os.path.dirname(code_filepath)
	# parent_dir after split: /user_codes/cpp/uuid
	# in sibling container, data is available at: $sibling_volume_mount/splited_parent_dir
	parent_dir = parent_dir.split(judge_volume_mount)[1]


View Diff
• Changing the import paths to relative imports may cause import errors if the module structure or execution context changes.

online-judge/src/core_apps/judge_engine/exec_engine.py

Lines 11 to 17 in 960a0f3

	# from core_apps.judge_engine.file_data_processor import file_processor
	# from core_apps.judge_engine.containers import code_container
	# from core_apps.judge_engine.compare_testcases import compare_output_and_testcases
	from file_data_processor import file_processor
	from containers import code_container
	from compare_testcases import compare_output_and_testcases


View Diff

src/core_apps/judge_engine/file_data_processor.py

• The additional return value judge_volume_mount in _process_write_data may cause issues if the caller does not handle the new tuple structure correctly.

online-judge/src/core_apps/judge_engine/file_data_processor.py

Lines 186 to 193 in 960a0f3

	True,
	code_filepath,
	input_filepath,
	output_filepath,
	testcases_filepath,
	success_message,
	judge_volume_mount,
	)


View Diff
• Sweep has identified a redundant function: The new function __get_user_code_base_dir is redundant because its functionality is already covered by the existing method __get_user_code_base_dir_with_lang in file_data_processor.py.

online-judge/src/core_apps/judge_engine/file_data_processor.py

Lines 18 to 42 in 960a0f3

	def __get_user_code_base_dir(self):
	"""Return the base-dir/user_codes directory
	Storage:
	volume:
	name: user_code_files
	The below '$base_dir' is the volume mount for the Judge Container.
	The volume mount of sibling container is specified at the module 'containers.py'.
	Hence, everything is saved under 'base_dir' in the Judge Container, will be available at
	the volume mount of sibling contianer.
	Ex: Judge Container file path: /app/user-files/user_codes/lang/uuid/
	Sibling Container file path for the same data: $volume_mount_of_sibling_container/user_codes/lang/uuid/
	"""
	# this is the volume for the main Judge Container. see the docker compose file.
	base_dir = "/app/user-files"
	logger.info(f"base dir: {base_dir}")
	user_code_dir = "user_codes"
	user_code_base_dir = os.path.join(base_dir, user_code_dir)
	return user_code_base_dir, base_dir


View Diff

src/core_apps/judge_engine/questions.py

• The function call sum_of_a in the C++ code should be sum_of_a_b, which will cause a compilation error.

online-judge/src/core_apps/judge_engine/questions.py

Line 3 in 960a0f3

"code": "#include <bits/stdc++.h>\nusing namespace std; \n\nvoid sum_of_a(int a, int b)\n{\n cout << a + b << endl; \n}\n\n\nint main()\n{\n int t, a, b; \n cin >> t; \n \n while(t--)\n {\n cin >> a >> b; \n sum_of_a_b(a, b); \n }\n\n return 0;\n}",


View Diff

src/core_apps/judge_engine/urls.py

• Commenting out the URL patterns and imports makes the "simple-execute/" and "robust-execute/" endpoints inaccessible, which could break functionality relying on these endpoints.

online-judge/src/core_apps/judge_engine/urls.py

Lines 4 to 17 in 960a0f3

	# from core_apps.judge_engine.views import CodeSubmitSimpleImplementation, CodeSubmitRobustAPI
	# urlpatterns = [
	# path(
	# "simple-execute/",
	# CodeSubmitSimpleImplementation.as_view(),
	# name="simple_execute_code",
	# ),
	# path("robust-execute/", CodeSubmitRobustAPI.as_view(), name="robust_execute_code"),
	# ]
	urlpatterns = [
	]


View Diff

src/core_apps/judge_engine/views.py

• Commenting out the CodeSubmitSimpleImplementation and CodeSubmitRobustAPI classes effectively disables the API endpoints, which could break functionality dependent on these endpoints.

online-judge/src/core_apps/judge_engine/views.py

Lines 47 to 149 in 960a0f3

	# class CodeSubmitSimpleImplementation(APIView):
	# """A simple approach to test the code submission without creating the files beforehand.
	# In this use case, the host volume creation also handled by docker.
	# The volume created in host by docker has permission only of docker, hence the files can not be deleted by host user.
	# The time is also same as the robust implementation of creating files beforehand. Infact, this method takes one second more
	# one average than the robust implementation.
	# """
	# def post(self, request):
	# start = time.time()
	# lang = request.data.get("lang")
	# code = request.data.get("code")
	# input_data = request.data.get("input")
	# print("Code: ", code)
	# client = from_env()
	# file_path = f"user-files/{random.randint(10, 100000)}"
	# curr_path = os.getcwd()
	# print("curr path: ", os.getcwd())
	# new_file_path = os.path.join(curr_path, file_path)
	# try:
	# # Create and start a container
	# container = client.containers.run(
	# "simple_cpp", # Image name
	# volumes={
	# f"{new_file_path}/": {
	# "bind": "/user-files",
	# "mode": "rw",
	# }
	# },
	# command=[
	# "sh",
	# "-c",
	# f'echo "{code}" > /user-files/code.cpp && echo "{input_data}" > /user-files/input.txt && g++ /user-files/code.cpp -o /user-files/code && /user-files/code < /user-files/input.txt > /user-files/output.txt',
	# ],
	# detach=True,
	# )
	# # Wait for the container to finish
	# result = container.wait()
	# print("result: ", result)
	# # Get the logs (output)
	# output = container.logs().decode("utf-8")
	# with open(f"{new_file_path}/output.txt", "r") as f:
	# data = f.read()
	# print("data: ", data)
	# # Remove the container
	# container.remove()
	# end = time.time()
	# print("total time taken: ", end - start)
	# # Return output to the user
	# return JsonResponse({"output": output})
	# except ContainerError as e:
	# print(f"Error: {e}")
	# return JsonResponse({"error": str(e)}, status=500)
	# except Exception as e:
	# print(f"Unexpected error: {e}")
	# return JsonResponse({"error": "An unexpected error occurred"}, status=500)
	# # return JsonResponse({"error": "Method not allowed"}, status=405)
	# class CodeSubmitRobustAPI(APIView):
	# """An API to test the robust implementation of the Online Judge.
	# and compare the time with simple implementation
	# """
	# def post(self, request):
	# """Submit code to execute in secure docker container."""
	# lang = request.data.get("lang")
	# code = request.data.get("code")
	# input_file = request.data.get("input")
	# testcases = request.data.get("testcases")
	# # print('lang: ', lang)
	# # print("code: ", code)
	# # print("input file ", input_file)
	# # print("testcases: ", testcases)
	# # print('request.data: ', request.data)
	# # print('\ntype of requst.data: ', type(request.data))
	# # print('\ntype of code: ', type(code))
	# # print('\ntype of input: ', type(input_file))
	# submission_id = uuid.uuid4()
	# data = code_exec_engine(user_codes=request.data, submission_id=submission_id)
	# return JsonResponse({"submission_id": submission_id, "data": data}, status=200)


View Diff
• The hardcoded problem definition and the immediate call to test_inside_cont_exe at the end of the file could lead to unintended execution and potential security risks if the script is run in a production environment.

online-judge/src/core_apps/judge_engine/views.py

Lines 28 to 164 in 960a0f3

	hello_world_with_num_data_problem = {
	"lang": "cpp",
	"code": '#include <bits/stdc++.h>\nusing namespace std;\n\nvoid sol(int num)\n{\n cout<<"Hello World: "<<num<<endl;\n}\n\nint main()\n{\n int t, num; \n cin>>t;\n\n while(t--)\n {\n cin>>num; \n sol(num);\n }\n\n return 0;\n}',
	"input": ["10", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10"],
	"testcases": [
	"Hello World: 1",
	"Hello World: 2",
	"Hello World: 3",
	"Hello World: 4",
	"Hello World: 5",
	"Hello World: 6",
	"Hello World: 7",
	"Hello World: 8",
	"Hello World: 9",
	"Hello World: 10",
	],
	}
	# class CodeSubmitSimpleImplementation(APIView):
	# """A simple approach to test the code submission without creating the files beforehand.
	# In this use case, the host volume creation also handled by docker.
	# The volume created in host by docker has permission only of docker, hence the files can not be deleted by host user.
	# The time is also same as the robust implementation of creating files beforehand. Infact, this method takes one second more
	# one average than the robust implementation.
	# """
	# def post(self, request):
	# start = time.time()
	# lang = request.data.get("lang")
	# code = request.data.get("code")
	# input_data = request.data.get("input")
	# print("Code: ", code)
	# client = from_env()
	# file_path = f"user-files/{random.randint(10, 100000)}"
	# curr_path = os.getcwd()
	# print("curr path: ", os.getcwd())
	# new_file_path = os.path.join(curr_path, file_path)
	# try:
	# # Create and start a container
	# container = client.containers.run(
	# "simple_cpp", # Image name
	# volumes={
	# f"{new_file_path}/": {
	# "bind": "/user-files",
	# "mode": "rw",
	# }
	# },
	# command=[
	# "sh",
	# "-c",
	# f'echo "{code}" > /user-files/code.cpp && echo "{input_data}" > /user-files/input.txt && g++ /user-files/code.cpp -o /user-files/code && /user-files/code < /user-files/input.txt > /user-files/output.txt',
	# ],
	# detach=True,
	# )
	# # Wait for the container to finish
	# result = container.wait()
	# print("result: ", result)
	# # Get the logs (output)
	# output = container.logs().decode("utf-8")
	# with open(f"{new_file_path}/output.txt", "r") as f:
	# data = f.read()
	# print("data: ", data)
	# # Remove the container
	# container.remove()
	# end = time.time()
	# print("total time taken: ", end - start)
	# # Return output to the user
	# return JsonResponse({"output": output})
	# except ContainerError as e:
	# print(f"Error: {e}")
	# return JsonResponse({"error": str(e)}, status=500)
	# except Exception as e:
	# print(f"Unexpected error: {e}")
	# return JsonResponse({"error": "An unexpected error occurred"}, status=500)
	# # return JsonResponse({"error": "Method not allowed"}, status=405)
	# class CodeSubmitRobustAPI(APIView):
	# """An API to test the robust implementation of the Online Judge.
	# and compare the time with simple implementation
	# """
	# def post(self, request):
	# """Submit code to execute in secure docker container."""
	# lang = request.data.get("lang")
	# code = request.data.get("code")
	# input_file = request.data.get("input")
	# testcases = request.data.get("testcases")
	# # print('lang: ', lang)
	# # print("code: ", code)
	# # print("input file ", input_file)
	# # print("testcases: ", testcases)
	# # print('request.data: ', request.data)
	# # print('\ntype of requst.data: ', type(request.data))
	# # print('\ntype of code: ', type(code))
	# # print('\ntype of input: ', type(input_file))
	# submission_id = uuid.uuid4()
	# data = code_exec_engine(user_codes=request.data, submission_id=submission_id)
	# return JsonResponse({"submission_id": submission_id, "data": data}, status=200)
	############################
	def test_inside_cont_exe():
	submission_id = uuid.uuid4()
	data = code_exec_engine(
	user_codes=hello_world_with_num_data_problem, submission_id=submission_id
	)
	print("data: \n", data)
	# This is the entrypoint function to check the Judge locally. No request is needed.
	test_inside_cont_exe()


View Diff
• Sweep has identified a redundant function: The new function test_inside_cont_exe is redundant because its functionality is already covered by the post method in the CodeSubmitRobustAPI class in src/core_apps/judge_engine/views.py.

online-judge/src/core_apps/judge_engine/views.py

Lines 154 to 160 in 960a0f3

	def test_inside_cont_exe():
	submission_id = uuid.uuid4()
	data = code_exec_engine(
	user_codes=hello_world_with_num_data_problem, submission_id=submission_id
	)
	print("data: \n", data)


View Diff

src/dev.yml

• The external volume user_code_files is assumed to exist, which may cause the service to fail if the volume is not created beforehand.

online-judge/src/dev.yml

Line 12 in 960a0f3

- user_code_files:/app/user-files


View Diff
• Mounting the Docker socket (/var/run/docker.sock) inside the container can pose a security risk as it allows the container to control the Docker daemon.

online-judge/src/dev.yml

Line 10 in 960a0f3

- /var/run/docker.sock:/var/run/docker.sock


View Diff

src/docker/dev/django/Dockerfile

• The line RUN curl -sSL https://get.docker.com/ | sh installs Docker inside the container, which is generally unnecessary and could introduce security vulnerabilities.

online-judge/src/docker/dev/django/Dockerfile

Line 46 in 960a0f3

RUN curl -sSL https://get.docker.com/ | sh


View Diff

src/docker/dev/django/Dockerfile-5

• The line to install dependencies using pip is commented out, which means the required packages will not be installed, potentially causing the application to fail to start.

online-judge/src/docker/dev/django/Dockerfile-5

Line 11 in 960a0f3

# RUN pip install --no-cache-dir -r ./requirements/dev.txt


View Diff

src/docker/dev/django/start

• The commented out set -o errexit, set -o pipefail, and set -o nounset options reduce the robustness of the script by not handling errors, pipeline failures, and unset variables.

online-judge/src/docker/dev/django/start

Lines 3 to 7 in 960a0f3

	# set -o errexit
	# set -o pipefail
	# set -o nounset


View Diff

src/docker/lang/cpp/script.sh

• The variable user_file_parent_dir is referenced but not defined, which will cause the script to fail.

online-judge/src/docker/lang/cpp/script.sh

Lines 10 to 12 in 960a0f3

	# user_file_parent_dir= is the full path of the file inside the volume: $volume_mount/user_codes/lang/uuid
	g++ $volume_mount$user_file_parent_dir/main.cpp -o $volume_mount$user_file_parent_dir/main


View Diff

---

Potential Issues

Sweep is unsure if these are issues, but they might be worth checking out.

src/core_apps/judge_engine/questions.py

• The expected test case results in the "testcases" key do not match the actual sums of the provided input pairs, which could lead to incorrect validation.

online-judge/src/core_apps/judge_engine/questions.py

Line 17 in 960a0f3

"testcases": ["3", "12", "10", "15", "30", "55", "200", "300", "200", "300"],


View Diff

---

[Mahboob-A]

Sweep: PR Review

Authors of pull request: @Mahboob-A

This pull request aimed to enhance the Docker-based development environment and refactor the code execution engine for better modularity and maintainability.

A Makefile was added to simplify Docker Compose commands, introducing dup and ddown targets for starting and stopping the development environment. The dev.yml Docker Compose file was created to define the services, networks, and volumes required for the development setup, including a judge service.

The containers.py file was updated to change the Docker image used for running containers and to persist containers by commenting out the stop and remove commands. The exec_engine.py and file_data_processor.py files were refactored to handle file paths and volume mounts more effectively, introducing variables like judge_volume_mount for better path management.

The views.py file saw significant changes, with the existing API views being commented out and replaced by a local testing function test_inside_cont_exe to facilitate local testing of the code execution engine.

New Dockerfiles were added for both the Django application and the C++ execution environment, with changes to the directory structure and volume mounts to ensure consistency across the development environment.

Overall, these changes aimed to streamline the development workflow, improve code modularity, and enhance the local testing capabilities of the code execution engine.

Sweep Found These Issues

Potential Issues

Ignore Sweep Issues. They are already covered.

[Mahboob-A]

This branch is only for running the Judge without calling the API which is available in the main branch.

Clone this branch and follow the steps described above to reproduce the desired Judge result.