Skip to content

feat(testset generation): implement new TestsetGenerator #487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jan 20, 2024
Merged

feat(testset generation): implement new TestsetGenerator #487

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1e74c58
delete old
jjmachan Jan 19, 2024
25cc7a5
added from_openai generator
jjmachan Jan 19, 2024
d3ae6e4
TestsetGenerator working
jjmachan Jan 19, 2024
ddf11b9
add benchmarks for testsetgen
jjmachan Jan 19, 2024
f71f402
utils to patch logger
jjmachan Jan 19, 2024
0113f7d
added logger
jjmachan Jan 20, 2024
5aeaa94
fmt
jjmachan Jan 20, 2024
05c579e
remove experiments
jjmachan Jan 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 14 additions & 9 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ format: ## Running code formatter: black and isort
@echo "(isort) Ordering imports..."
@isort .
@echo "(black) Formatting codebase..."
@black --config pyproject.toml src tests docs experiments
@black --config pyproject.toml src tests docs
@echo "(black) Formatting stubs..."
@find src -name "*.pyi" ! -name "*_pb2*" -exec black --pyi --config pyproject.toml {} \;
@echo "(ruff) Running fix only..."
Expand All @@ -25,14 +25,6 @@ clean: ## Clean all generated files
@cd $(GIT_ROOT) || exit 1
@find . -type f -name '*.py[co]' -delete -o -type d -name __pycache__ -delete
run-ci: format lint type ## Running all CI checks
run-benchmarks: ## Run benchmarks
@echo "Running benchmarks..."
@cd $(GIT_ROOT)/tests/benchmarks && python benchmark_eval.py
run-benchmarks-in-docker: ## Run benchmarks in docker
@echo "Running benchmarks in docker..."
@cd $(GIT_ROOT)
docker buildx build --build-arg OPENAI_API_KEY=$(OPENAI_API_KEY) -t ragas-benchmark -f $(GIT_ROOT)/tests/benchmarks/Dockerfile .
docker inspect ragas-benchmark:latest | jq ".[0].Size" | numfmt --to=si
test: ## Run tests
@echo "Running tests..."
@pytest tests/unit $(shell if [ -n "$(k)" ]; then echo "-k $(k)"; fi)
Expand All @@ -46,3 +38,16 @@ docs-site: ## Build and serve documentation
@python -m http.server --directory $(GIT_ROOT)/docs/_build/html
watch-docs: ## Build and watch documentation
sphinx-autobuild docs docs/_build/html --watch $(GIT_ROOT)/src/ --ignore ".ipynb"

# Benchmarks
run-benchmarks-eval: ## Run benchmarks for Evaluation
@echo "Running benchmarks for Evaluation..."
@cd $(GIT_ROOT)/tests/benchmarks && python benchmark_eval.py
run-benchmarks-testset: ## Run benchmarks for TestSet Generation
@echo "Running benchmarks for TestSet Generation..."
@cd $(GIT_ROOT)/tests/benchmarks && python benchmark_testsetgen.py
run-benchmarks-in-docker: ## Run benchmarks in docker
@echo "Running benchmarks in docker..."
@cd $(GIT_ROOT)
docker buildx build --build-arg OPENAI_API_KEY=$(OPENAI_API_KEY) -t ragas-benchmark -f $(GIT_ROOT)/tests/benchmarks/Dockerfile .
docker inspect ragas-benchmark:latest | jq ".[0].Size" | numfmt --to=si
Loading