Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,48 @@ $env:OPENML_TEST_SERVER_ADMIN_KEY = "admin-key"
export OPENML_TEST_SERVER_ADMIN_KEY="admin-key"
```

#### Running Tests Against Local Services (Docker)

The CI now runs tests against a local OpenML server using Docker instead of the remote test server.
You can replicate this setup locally:

1. **Clone the services repository** (from the parent directory of `openml-python`):
```bash
git clone --depth 1 https://github.com/openml/services.git
cd services
```

2. **Set file permissions** required by the containers:
```bash
chmod -R a+rw ./data
chmod -R a+rw ./logs
```

3. **Start the Docker services**:
```bash
docker compose --profile rest-api --profile minio --profile evaluation-engine up -d
```

4. **Wait for the API to become healthy**:
```bash
timeout 60s bash -c 'until [ "$(docker inspect -f {{.State.Health.Status}} openml-php-rest-api)" == "healthy" ]; do sleep 5; done'
```

5. **Run the tests** with the local services environment variable:
```bash
export OPENML_USE_LOCAL_SERVICES="true"
pytest -n 4 --durations=20 --dist load -sv -m "not production_server"
```

6. **Cleanup** after testing:
```bash
cd services
docker compose --profile rest-api --profile minio --profile evaluation-engine down
```

For more details, see the [Developer Setup Guide](https://openml.github.io/openml-python/main/developer_setup.html).


### Pull Request Checklist

You can go to the `openml-python` GitHub repository to create the pull request by [comparing the branch](https://github.com/openml/openml-python/compare) from your fork with the `main` branch of the `openml-python` repository. When creating a pull request, make sure to follow the comments and structured provided by the template on GitHub.
Expand Down
78 changes: 78 additions & 0 deletions docs/developer_setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,83 @@ Exclude production tests (local only):
pytest -m "not production_server"
```

### Running Tests Against Local Services

The CI pipeline runs tests against a local OpenML server using Docker, instead of the remote
test server. You can replicate this setup locally to avoid depending on the remote server.

#### Prerequisites

* **Docker**: Docker Desktop (ensure the daemon is running).
* **Git**: To clone the services repository.

#### 1. Clone the Services Repository

```bash
git clone --depth 1 https://github.com/openml/services.git
cd services
```

#### 2. Configure File Permissions

The containerized services need write access to the data and log directories:

```bash
chmod -R a+rw ./data
chmod -R a+rw ./logs
```

#### 3. Start the Docker Services

Launch the required service profiles (REST API, MinIO, and Evaluation Engine):

```bash
docker compose --profile rest-api --profile minio --profile evaluation-engine up -d
```

Wait for the PHP API to become healthy:

```bash
# Wait up to 60 seconds for the API to boot
timeout 60s bash -c 'until [ "$(docker inspect -f {{.State.Health.Status}} openml-php-rest-api)" == "healthy" ]; do sleep 5; done'
```

Verify the services are running:

```bash
curl -sSfL http://localhost:8000/api/v1/xml/data/1 | head -n 15
```

#### 4. Run Tests Against Local Services

Set the `OPENML_USE_LOCAL_SERVICES` environment variable and run pytest:

=== "Linux/macOS"

```bash
export OPENML_USE_LOCAL_SERVICES="true"
pytest -n 4 --durations=20 --dist load -sv -m "not production_server"
```

=== "Windows (PowerShell)"

```shell
$env:OPENML_USE_LOCAL_SERVICES = "true"
pytest -n 4 --durations=20 --dist load -sv -m "not production_server"
```

When this environment variable is set, the test framework automatically redirects API calls
to `http://localhost:8000` instead of the remote test server.

#### 5. Cleanup

After you're done testing, stop and remove the Docker containers:

```bash
cd services
docker compose --profile rest-api --profile minio --profile evaluation-engine down
```

### Admin Privilege Tests

Certain tests require administrative privileges on the test server. These are skipped automatically unless an admin API key is provided via environment variables.
Expand All @@ -208,3 +285,4 @@ $env:OPENML_TEST_SERVER_ADMIN_KEY = "admin-key"
```bash
export OPENML_TEST_SERVER_ADMIN_KEY="admin-key"
```