Updated relevant env for better testing

scripts/exp/ablation/env/basic.env

@@ -1,5 +1,3 @@
 KG_IF_USING_VECTOR_RAG=False
 KG_IF_USING_GRAPH_RAG=False
-KG_IF_ACTION_CHOOSING_BASED_ON_UCB=False
-KG_MODEL_FEATURE_SELECTION_CODER=True
-KG_HYPOTHESIS_GEN=True
+KG_IF_ACTION_CHOOSING_BASED_ON_UCB=False

scripts/exp/ablation/env/max.env

@@ -1,5 +1,3 @@
 KG_IF_USING_VECTOR_RAG=True
 KG_IF_USING_GRAPH_RAG=True
-KG_IF_ACTION_CHOOSING_BASED_ON_UCB=True
-KG_MODEL_FEATURE_SELECTION_CODER=True
-KG_HYPOTHESIS_GEN=True 
+KG_IF_ACTION_CHOOSING_BASED_ON_UCB=True

scripts/exp/ablation/env/mini-case.env

@@ -1,5 +1,3 @@
 KG_IF_USING_VECTOR_RAG=True
 KG_IF_USING_GRAPH_RAG=False
 KG_IF_ACTION_CHOOSING_BASED_ON_UCB=True
-KG_MODEL_FEATURE_SELECTION_CODER=True
-KG_HYPOTHESIS_GEN=False 

scripts/exp/ablation/env/pro.env

@@ -1,5 +1,3 @@
 KG_IF_USING_VECTOR_RAG=True
 KG_IF_USING_GRAPH_RAG=False
 KG_IF_ACTION_CHOOSING_BASED_ON_UCB=True
-KG_MODEL_FEATURE_SELECTION_CODER=True
-KG_HYPOTHESIS_GEN=True 

scripts/exp/tools/README.md

@@ -1,3 +1,138 @@
-The tools in the directory contains following generalfeatures
-- collecting envs and run each
-- collect results and generate summary
+Tools Directory
+
+This directory provides scripts to run experiments with different environment configurations, collect results, and demonstrate usage through an example script.
+
+Directory Structure
+
+scripts/exp/tools/
+├── run_envs.sh       # Script for running experiments
+├── collect.py        # Results collection and summary
+├── test_system.sh    # Example usage script
+├── README.md         # This documentation
+
+Tools Overview
+
+	1.	run_envs.sh: Executes experiments with different environment configurations in parallel.
+	2.	collect.py: Collects and summarizes experiment results into a single file.
+	3.	test_system.sh: Demonstrates how to use the above tools together for experiment execution and result collection.
+
+Getting Started
+
+Prerequisites
+
+	1.	Ensure the scripts have execution permissions:
+
+chmod +x scripts/exp/tools/run_envs.sh
+chmod +x scripts/exp/tools/test_system.sh
+
+
+	2.	Install required dependencies for Python scripts:
+
+pip install -r requirements.txt
+
+
+	3.	Place your .env files in the desired directory for environment configurations.
+
+Usage
+
+1. Running Experiments with Different Environments
+
+The run_envs.sh script allows running a command with multiple environment configurations in parallel.
+
+Command Syntax
+
+./run_envs.sh -d <dir_to_.envfiles> -j <number_of_parallel_processes> -- <command>
+
+Example Usage
+
+Basic example:
+
+./run_envs.sh -d env_files -j 1 -- echo "Hello"
+
+Practical example:
+
+dotenv run -- ./run_envs.sh -d /h/home/v-xisenwang/RD-Agent/scripts/exp/ablation/env -j 1 -- python /home/v-xisenwang/RD-Agent/rdagent/app/kaggle/loop.py
+
+Explanation:
+	•	-d: Specifies the directory containing .env files.
+	•	-j: Number of parallel processes to run (e.g., 1 for sequential execution).
+	•	--: Separates script options from the command to execute.
+	•	<command>: The command to execute with the environment variables loaded.
+
+2. Collecting Results
+
+The collect.py script processes logs and generates a summary JSON file.
+
+Command Syntax
+
+python collect.py --log_path <path_to_logs> --output_name <summary_filename>
+
+Example Usage
+
+Collect results from logs:
+
+python collect.py --log_path logs --output_name summary.json
+
+Explanation:
+	•	--log_path: Required. Specifies the directory containing experiment logs.
+	•	--output_name: Optional. The name of the output summary file (default: summary.json).
+
+3. Example Workflow
+
+Use the test_system.sh script to demonstrate a complete workflow.
+
+Steps:
+
+	1.	Ensure the scripts are executable:
+
+chmod +x scripts/exp/tools/run_envs.sh
+chmod +x scripts/exp/tools/test_system.sh
+
+
+	2.	Run the test system:
+
+./scripts/exp/tools/test_system.sh
+
+
+
+This will:
+	1.	Load environment configurations from .env files.
+	2.	Execute experiments using the configurations.
+	3.	Collect and summarize results.
+
+Troubleshooting
+
+Permission Denied
+
+If you encounter a PermissionError when running scripts:
+	1.	Ensure the script has execution permissions:
+
+chmod +x ./scripts/exp/tools/run_envs.sh
+chmod +x ./scripts/exp/tools/test_system.sh
+
+
+	2.	Verify file ownership:
+
+ls -l ./scripts/exp/tools/
+
+
+	3.	Change ownership or update permissions if necessary:
+
+sudo chown $USER:$USER ./scripts/exp/tools/*
+chmod 755 ./scripts/exp/tools/*
+
+Capitalized Environment Variables
+
+To standardize variable names, .env files should use uppercase variable names with underscores (e.g., MY_VARIABLE). For example:
+
+IF_USING_VECTOR_RAG=true
+IF_USING_GRAPH_RAG=false
+MODEL_FEATURE_SELECTION_CODER=advanced
+
+Notes
+
+	•	Use the -j parameter to scale parallel processes as needed.
+	•	Ensure .env files are correctly formatted to avoid errors.
+	•	Customize test_system.sh as per your project’s requirements.
+
+For further assistance, refer to the comments within the scripts or reach out to the development team.