-
-
Notifications
You must be signed in to change notification settings - Fork 156
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Enabling automation of experiments running v2.0 #469
Open
xisen-w
wants to merge
24
commits into
main
Choose a base branch
from
automated-evaluation
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+341
−0
Open
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
8726ab8
Revising to enable automation of experiments running v1.0
xisen-w b44bef5
Any new updates
xisen-w c100876
Revising to enable automation of experiments running v1.0
xisen-w 18370d4
Any new updates
xisen-w 21a99d2
Add template
you-n-g 86ae0b2
Stoping tracking additional env
xisen-w f94dbff
Merge branch 'automated-evaluation' of https://github.com/microsoft/R…
xisen-w 66ffd6d
Uploading relevant envs
xisen-w 0ef80a5
Adding tests
xisen-w 907d980
Updating
xisen-w 51388d1
Updated collect.py to extract result from trace
xisen-w af6220e
Update .gitignore to remove the unecessary ones
xisen-w 54c3c6d
"Remove unnecessary files"
xisen-w 78708e4
Merge branch 'automated-evaluation' of https://github.com/microsoft/R…
xisen-w 3f131f3
Merge branch 'main' into automated-evaluation
xisen-w 38bb9e6
Updated to enable automatic collection of experiment result information
xisen-w 10b0053
Updating the env files & Upading test_system file
xisen-w 238f492
Updated relevant env for better testing
xisen-w 68ca63a
Updated README.md
xisen-w 8b18fad
reverting gitignore back
xisen-w 2395dc5
Updates
xisen-w b7cc98e
README update
xisen-w 0b5a09d
Updates on env README
xisen-w 24cd0c2
Updating collect.py
xisen-w File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Add template
- Loading branch information
commit 21a99d27de948a9a3dc527c24dd73862c3a74c99
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# Introduction | ||
|
||
| name | .env | desc | | ||
| -- | -- | -- | | ||
| full | full.env | enable all features | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
The tools in the directory contains following generalfeatures | ||
- collecting envs and run each | ||
- collect results and generate summary |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
#!/bin/sh | ||
cat << "EOF" > /dev/null | ||
Given a directory with *.env files. Run each one. | ||
usage for example: | ||
1) directly run command without extra shared envs | ||
./run_envs.sh -d <dir_to_*.envfiles> -j <number of parallel process> -- <command> | ||
2) load shared envs `.env` before running command with different envs. | ||
dotenv run -- ./run_envs.sh -d <dir_to_*.envfiles> -j <number of parallel process> -- <command> | ||
EOF | ||
|
||
# Function to display usage | ||
usage() { | ||
echo "Usage: $0 -d <dir_to_*.envfiles> -j <number of parallel process> -- <command>" | ||
exit 1 | ||
} | ||
|
||
# Parse command line arguments | ||
while getopts "d:j:" opt; do | ||
case $opt in | ||
d) DIR=$OPTARG ;; | ||
j) JOBS=$OPTARG ;; | ||
*) usage ;; | ||
esac | ||
done | ||
|
||
# Shift to get the command | ||
shift $((OPTIND -1)) | ||
|
||
# Check if directory and jobs are set | ||
if [ -z "$DIR" ] || [ -z "$OBS" ] || [ $# -eq 0 ]; then | ||
usage | ||
fi | ||
|
||
COMMAND="$@" | ||
|
||
# Export and run each .env file in parallel | ||
find "$DIR" -name "*.env" | xargs -n 1 -P "$JOBS" -I {} sh -c " | ||
set -a | ||
. {} | ||
set +a | ||
$COMMAND | ||
" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this file is not complete