-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve dataset generation #3
base: main
Are you sure you want to change the base?
Conversation
Dataset is now generated by posting parameters to datasim servers and reading the response.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -0,0 +1 @@ | |||
names==0.3.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would recommend to use Faker instead (see the person provider).
@@ -16,6 +16,8 @@ else | |||
exit 1 | |||
fi | |||
fi | |||
# Ignore docker orphans warning | |||
export COMPOSE_IGNORE_ORPHANS=True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be better to clean your environment instead of enforcing it in this (dead?) code.
# Run datasim server | ||
DOCKER_USER=${DOCKER_USER} ${COMPOSE} \ | ||
-f docker-compose.yml \ | ||
up -d datasim | ||
DOCKER_USER=${DOCKER_USER} ${COMPOSE} \ | ||
run --rm dockerize \ | ||
-wait http://datasim:9090/health \ | ||
-timeout 60s |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO this logic should stand in the Makefile (and in the docker-compose
file service dependency).
ports: | ||
- "8089:8089" | ||
working_dir: /app | ||
# network_mode: host |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see 💀 code here 😅
|
||
# Request datasim server a dataset generation | ||
resp = requests.post( | ||
"http://datasim:9090/api/v1/generate", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Datasim server endpoint should be configurable.
Dataset now generated by posting parameters to datasim servers and reading the response.
For now, we request datasim before each locust request, changing the seed between each dataset generation to have a new statement.
We need to measure how much time and resource requests to datasim server are taking and ensure it is acceptable, else we would need to generate dataset before a run.