Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve dataset generation #3

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Improve dataset generation #3

wants to merge 1 commit into from

Conversation

wilbrdt
Copy link
Contributor

@wilbrdt wilbrdt commented Apr 25, 2023

Dataset now generated by posting parameters to datasim servers and reading the response.

For now, we request datasim before each locust request, changing the seed between each dataset generation to have a new statement.
We need to measure how much time and resource requests to datasim server are taking and ensure it is acceptable, else we would need to generate dataset before a run.

Dataset is now generated by posting parameters to datasim servers and reading
the response.
@wilbrdt wilbrdt self-assigned this Apr 25, 2023
Copy link
Contributor

@jmaupetit jmaupetit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -0,0 +1 @@
names==0.3.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend to use Faker instead (see the person provider).

@@ -16,6 +16,8 @@ else
exit 1
fi
fi
# Ignore docker orphans warning
export COMPOSE_IGNORE_ORPHANS=True
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to clean your environment instead of enforcing it in this (dead?) code.

Comment on lines +51 to +58
# Run datasim server
DOCKER_USER=${DOCKER_USER} ${COMPOSE} \
-f docker-compose.yml \
up -d datasim
DOCKER_USER=${DOCKER_USER} ${COMPOSE} \
run --rm dockerize \
-wait http://datasim:9090/health \
-timeout 60s
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO this logic should stand in the Makefile (and in the docker-compose file service dependency).

ports:
- "8089:8089"
working_dir: /app
# network_mode: host
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see 💀 code here 😅


# Request datasim server a dataset generation
resp = requests.post(
"http://datasim:9090/api/v1/generate",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datasim server endpoint should be configurable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants