Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLI #471

Merged
merged 44 commits into from
Mar 25, 2024
Merged

Add CLI #471

merged 44 commits into from
Mar 25, 2024

Conversation

gabrielmbmb
Copy link
Member

Description

This PR adds a CLI application to distilabel that allows (for now) to:

  1. Run a pipeline given the config YAML file that can be either a local file or a URL:
    distilabel pipeline run --config pipeline.yaml \
        --param load_hub_dataset.repo_id=HuggingFaceH4/instruction-dataset \
        --param load_hub_dataset.split=test \
        --param text_generation_gpt.num_generations=3 \
        --param text_generation_gpt_2.num_generations=3 \
        --param push_to_hub.repo_id=distilabel-internal-testing/testing-distilabel-push-to-hub \
        --param push_to_hub_2.repo_id=distilabel-internal-testing/testing-distilabel-push-to-hub-2 \
        --ignore-cache \
        --repo-id distilabel-internal-testing/testing-distilabel-cli \
        --commit-message "Testing pipeline"
  2. Get the information of a pipeline (name, description, steps, runtime parameters, etc) given the config YAML file or a URL:
    distilabel pipeline info --config pipeline.yaml

@gabrielmbmb gabrielmbmb requested review from plaguss and alvarobartt and removed request for plaguss March 25, 2024 11:15
@gabrielmbmb gabrielmbmb self-assigned this Mar 25, 2024
@gabrielmbmb gabrielmbmb added the enhancement New feature or request label Mar 25, 2024
@gabrielmbmb gabrielmbmb linked an issue Mar 25, 2024 that may be closed by this pull request
@gabrielmbmb gabrielmbmb marked this pull request as ready for review March 25, 2024 11:15
@alvarobartt alvarobartt added this to the 1.0.0 milestone Mar 25, 2024
pyproject.toml Outdated Show resolved Hide resolved
pyproject.toml Show resolved Hide resolved
src/distilabel/pipeline/base.py Outdated Show resolved Hide resolved
src/distilabel/pipeline/local.py Outdated Show resolved Hide resolved
Co-authored-by: Agus <agustin@argilla.io>
@gabrielmbmb gabrielmbmb merged commit 160e6f6 into core-refactor Mar 25, 2024
4 checks passed
@gabrielmbmb gabrielmbmb deleted the cli branch March 25, 2024 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

CLI with run command
3 participants