diff --git a/README.md b/README.md index bea4f2e2..66dfb993 100644 --- a/README.md +++ b/README.md @@ -1,125 +1,101 @@ # Mint [![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/mint.svg?maxAge=604800)](https://hub.docker.com/r/minio/mint/) -Mint detects overall correctness of Minio server using Minio SDKs, minio client (mc), AWS SDKs, awscli and s3cmd. Use Mint to test Minio server instances for functional correctness. +Mint is a testing framework for Minio object server, available as a docker image. It runs correctness, benchmarking and stress tests. Following are the SDKs/tools used in correctness tests. -## Get started +- awscli +- aws-sdk-php +- aws-sdk-ruby +- mc +- minio-go +- minio-java +- minio-js +- minio-py +- s3cmd + +## Running Mint -Mint is available as a Docker image on official Minio Docker repository. You'll need Docker installed on your host before running Mint. Install and setup Docker by following the steps [here](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/). +Mint is run by `docker run` command which requires Docker to be installed. For Docker installation follow the steps [here](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/). -To run Mint image, use the `docker run` command. For example, to run Mint with Minio Play server as test target use the below command +To run Mint with Minio Play server as test target, ```sh -$ docker run -e SERVER_ENDPOINT=play.minio.io:9000 -e ACCESS_KEY=Q3AM3UQ867SPQQA43P2F \ --e SECRET_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG -e ENABLE_HTTPS=1 minio/mint:latest +$ docker run -e SERVER_ENDPOINT=play.minio.io:9000 -e ACCESS_KEY=Q3AM3UQ867SPQQA43P2F \ + -e SECRET_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG -e ENABLE_HTTPS=1 minio/mint ``` -After the tests are run, output is stored in `/mint/log` directory inside the container. You can access these logs via `docker cp` command. For example to store logs to `/tmp/logs` directory on your host, run - +After the tests are run, output is stored in `/mint/log` directory inside the container. To get these logs, use `docker cp` command. For example ```sh docker cp :/mint/log /tmp/logs ``` -Then navigate to `/tmp/logs` directory to access the test logs. +### Mint environment variables -#### Mint log format +Below environment variables are required to be passed to the docker container. Supported environment variables: -Mint generates logs in `json` format. Logs are aggregated in one file, and available at `/mint/log/log.json`. Below is a sample Mint log snippet +| Environment variable | Description | Example | +|:--- |:--- |:--- | +| `SERVER_ENDPOINT` | Endpoint of Minio server in the format `HOST:PORT` | `play.minio.io:9000` | +| `ACCESS_KEY` | Access key of access `SERVER_ENDPOINT` | `Q3AM3UQ867SPQQA43P2F` | +| `SECRET_KEY` | Secret Key of access `SERVER_ENDPOINT` | `zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG` | +| `ENABLE_HTTPS` | (Optional) Set `1` to indicate to use HTTPS to access `SERVER_ENDPOINT`. Defaults to `0` (HTTP) | `1` | +| `MINT_MODE` | (Optional) Set mode indicating what catogery of tests to be run by values `core` or `full`. Defaults to `core` | `full` | -```json -{ -"name":"", -"function":"", -"args":"", -"duration":"", -"status":"", -"alert":"", -"message":"", -"error":"" -} -``` +### Mint log format -Section wise description of Mint log fields +All test logs are stored in `/mint/log/log.json` as multiple JSON document. Below is the JSON format for every entry in the log file. -| File name | Size -|:--- |:--- | -|`name`| Name of SDK being tested | -|`function`| API signature being tested by the test case | -|`args`| Key value map of arguments passed in the API being tested. Note that this is not exhaustive, and only significant arguments will be listed | -|`duration`| Duration of the test in milliseconds | -|`status`| PASS, FAIL, NA | -|`alert`| Important information related to test failure. This should be taken care urgently if present | -|`message`| Textual description of the error | -|`error`| Actual exception/error thrown by the program | +| JSON field | Type | Description | Example | +|:--- |:--- |:--- |:--- | +| `name` | _string_ | Testing tool/SDK name | `"aws-sdk-php"` | +| `function` | _string_ | Test function name | `"getBucketLocation ( array $params = [] )"` | +| `args` | _object_ | (Optional) Key/Value map of arguments passed to test function | `{"Bucket":"aws-sdk-php-bucket-20341"}` | +| `duration` | _int_ | Time taken in milliseconds to run the test | `384` | +| `status` | _string_ | one of `PASS`, `FAIL` or `NA` | `"PASS"` | +| `alert` | _string_ | (Optional) Alert message indicating test failure | `"I/O error on create file"` | +| `message` | _string_ | (Optional) Any log message | `"validating checksum of downloaded object"` | +| `error` | _string_ | Detailed error message including stack trace on status `FAIL` | `"Error executing \"CompleteMultipartUpload\" on ...` | -#### Mint environment variables +## For Developers -Set environment variables to pass test target server details to the docker container. Supported environment variables: +### Running Mint development code -- `SERVER_ENDPOINT`- : of the Minio server on which the tests has to be run. Defaults to [Minio Play Server](play.minio.io:9000/minio/). -- `ACCESS_KEY` - Access Key of the server. Defaults to Minio Play Access Key. -- `SECRET_KEY` - Secret Key of the server. Defaults to Minio Play Secret Key. -- `ENABLE_HTTPS` - Set to 1 to send HTTPS requests on SSL enabled deployment. Defaults to 0. -- `MINT_DATA_DIR` - Data directory for SDK tests. Defaults to `/mint/data/`. +After making changes to Mint source code a local docker image can be built/run by -#### Current tests +```sh +$ docker build -t minio/mint . -f Dockerfile.dev +$ docker run -e SERVER_ENDPOINT=play.minio.io:9000 -e ACCESS_KEY=Q3AM3UQ867SPQQA43P2F \ + -e SECRET_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG -e ENABLE_HTTPS=1 \ + -e MINT_MODE=full minio/mint:latest +``` -Following SDKs/CLI tools are available in Mint: +### Adding tests with new tool/SDK -- awscli -- aws-sdk-php -- aws-sdk-ruby -- mc -- minio-go -- minio-java -- minio-js -- minio-py -- s3cmd +Below are the steps need to be followed + +* Create new app directory under [build](https://github.com/minio/mint/tree/master/build) and [run/core](https://github.com/minio/mint/tree/master/run/core) directories. +* Create `install.sh` which does installation of required tool/SDK under app directory. +* Any build and install time dependencies should be added to [install-packages.list](https://github.com/minio/mint/blob/master/install-packages.list). +* Build time dependencies should be added to [remove-packages.list](https://github.com/minio/mint/blob/master/remove-packages.list) for removal to have clean Mint docker image. +* Add `run.sh` in app directory under `run/core` which execute actual tests. #### Test data -All test data used by SDK tests reside in `/mint/data/` directory on the container. These are the files currently available in Mint data directory: +Tests may use pre-created data set to perform various object operations on Minio server. Below data files are available under `/mint/data` directory. -| File name | Size +| File name | Size | |:--- |:--- | | datafile-1-b | 1B | -| datafile-10-kB |10KB -| datafile-33-kB |33KB -| datafile-100-kB |100KB -| datafile-1-MB |1MB -| datafile-1.03-MB |1.03MB -| datafile-5-MB |5MB -| datafile-6-MB |6MB -| datafile-11-MB |11MB -| datafile-65-MB |65MB - -## FAQs - -#### How to add tests to Mint? - -To add tests to an existing SDK folder: - -- Add tests to respective SDK repository functional test. - -To add new SDK/CLI tool to Mint: - -- Check if the environment for the programming language is already set in `build` directory, if not, add a new directory for the language and add set up steps (including SDK/CLI tool) in `build//install.sh` directory. -- Create new directory in `run/core/` directory with corresponding SDK/CLI tool name. -- Add a `run.sh` script. This script should set up the SDK/CLI tool and then execute the tests - -#### How to build Mint image locally? - -```sh -$ git clone https://github.com/minio/mint.git -$ cd mint -$ docker build -t minio/mint . -f Dockerfile.dev -``` - -Developers can also customize `Dockerfile.dev` to generate smaller build images. For example, removing the following lines from `Dockerfile.dev` will avoid shipping Golang SDK tests in the image hence a faster build and tests execution time. - -```docker -COPY build/go/ /mint/build/go/ -RUN /mint/build/go/install.sh -``` +| datafile-10-kB |10KiB | +| datafile-33-kB |33KiB | +| datafile-100-kB |100KiB | +| datafile-1-MB |1MiB | +| datafile-1.03-MB |1.03MiB | +| datafile-5-MB |5MiB | +| datafile-6-MB |6MiB | +| datafile-11-MB |11MiB | +| datafile-65-MB |65MiB | + +#### Mint image of github pull request **Note for Developers**: On each PR sent to [Mint repository](https://github.com/minio/mint), `travis-ci` builds `mint` docker image and pushes it to `play.minio.io`, our private docker registry. You can get the `mint` image associated with your pull request by just running `docker pull play.minio.io/mint:$PULL_REQUEST_SHA`. For example