Skip to content

Commit

Permalink
Update README sections and rules planned for addition (#144)
Browse files Browse the repository at this point in the history
  • Loading branch information
SmetDenis committed Apr 4, 2024
1 parent ef6f379 commit a8087c2
Show file tree
Hide file tree
Showing 7 changed files with 239 additions and 215 deletions.
87 changes: 45 additions & 42 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,45 +59,48 @@ jobs:
run: make bench-create-csv bench-docker --no-print-directory


php-benchmark:
name: Phar
runs-on: ubuntu-latest
strategy:
matrix:
version: [ latest, local ]
columns: [ 1, 5, 10, 20 ]
env:
BENCH_COLS: ${{ matrix.columns }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
ref: ${{ github.event.inputs.branch || github.ref }}

- name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: 8.3
coverage: none
ini-values: opcache.enable_cli=1, opcache.jit=1255

- name: Build local Phar file
if: matrix.version == 'local'
run: |
make build-prod build-phar-file --no-print-directory
./build/csv-blueprint.phar
- name: Download latest Phar file
if: matrix.version == 'latest'
run: |
wget https://github.com/JBZoo/Csv-Blueprint/releases/latest/download/csv-blueprint.phar -O ./build/csv-blueprint.phar
chmod +x ./build/csv-blueprint.phar
./build/csv-blueprint.phar
# We need it to build benchmark tool. See ./tests/Benchmarks
- name: Build project
run: make build --no-print-directory

- name: 🔥 Benchmark 🔥
run: make bench-create-csv bench-phar --no-print-directory
# No reason to use Phar anymore. Docker is much better and faster.
# Keep it here just for quick benchmarking on demand.
#
# php-benchmark:
# name: Phar
# runs-on: ubuntu-latest
# strategy:
# matrix:
# version: [ latest, local ]
# columns: [ 1, 5, 10, 20 ]
# env:
# BENCH_COLS: ${{ matrix.columns }}
# steps:
# - name: Checkout code
# uses: actions/checkout@v4
# with:
# fetch-depth: 0
# ref: ${{ github.event.inputs.branch || github.ref }}
#
# - name: Setup PHP
# uses: shivammathur/setup-php@v2
# with:
# php-version: 8.3
# coverage: none
# ini-values: opcache.enable_cli=1, opcache.jit=1255
#
# - name: Build local Phar file
# if: matrix.version == 'local'
# run: |
# make build-prod build-phar-file --no-print-directory
# ./build/csv-blueprint.phar
#
# - name: Download latest Phar file
# if: matrix.version == 'latest'
# run: |
# wget https://github.com/JBZoo/Csv-Blueprint/releases/latest/download/csv-blueprint.phar -O ./build/csv-blueprint.phar
# chmod +x ./build/csv-blueprint.phar
# ./build/csv-blueprint.phar
#
# # We need it to build benchmark tool. See ./tests/Benchmarks
# - name: Build project
# run: make build --no-print-directory
#
# - name: 🔥 Benchmark 🔥
# run: make bench-create-csv bench-phar --no-print-directory
5 changes: 5 additions & 0 deletions .github/workflows/demo.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ name: Demo
on:
release:
types: [ created ]
workflow_dispatch:
inputs:
branch:
description: 'Branch Name'
required: false
workflow_run:
workflows: [ "Publish Docker" ]
types:
Expand Down
59 changes: 36 additions & 23 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,28 +22,20 @@ on:
schedule:
- cron: '0 0 * * *'

# We want to avoid copy-pasting the same commands and add control over the environment

# To avoid copy-pasting the same commands and add control over the environment
env:
COLUMNS: 300
BLUEPRINT: time ./csv-blueprint
BLUEPRINT_PHAR: time ./build/csv-blueprint.phar
BLUEPRINT_DOCKER: |
time docker run --rm
--workdir=/parent-host
-v .:/parent-host
jbzoo/csv-blueprint:local
BLUEPRINT_DOCKER: time docker run --rm --workdir=/parent-host -v .:/parent-host jbzoo/csv-blueprint:local
CMD_VALIDATE: validate:csv --ansi -vvv
VALID_TEST: |
--csv=./tests/fixtures/batch/*.csv
--schema=./tests/schemas/demo_valid.yml
INVALID_TEST: |
--csv=./tests/fixtures/batch/*.csv
--schema=./tests/schemas/demo_*.yml
--schema=./tests/schemas/invalid_schema.yml
VALID_TEST: --csv=./tests/fixtures/batch/*.csv --schema=./tests/schemas/demo_valid.yml
INVALID_TEST: --csv=./tests/fixtures/batch/*.csv --schema=./tests/schemas/demo_*.yml --schema=./tests/schemas/invalid_schema.yml

jobs:
test-current-versions:
name: Tests - Current
name: Tests
runs-on: ubuntu-latest
steps:
- name: Checkout code
Expand All @@ -67,9 +59,6 @@ jobs:
- name: 👍 Code Quality
run: make codestyle --no-print-directory

- name: 📝 Build Reports
run: make report-all --no-print-directory

- name: Uploading coverage to coveralls
continue-on-error: true
env:
Expand Down Expand Up @@ -115,9 +104,6 @@ jobs:
- name: 👍 Code Quality
run: make codestyle --no-print-directory

- name: 📝 Build Reports
run: make report-all --no-print-directory

- name: Upload Artifacts
uses: actions/upload-artifact@v4
continue-on-error: true
Expand Down Expand Up @@ -157,9 +143,6 @@ jobs:
- name: 👍 Code Quality
run: make codestyle --no-print-directory

- name: 📝 Build Reports
run: make report-all --no-print-directory

- name: Upload Artifacts
uses: actions/upload-artifact@v4
continue-on-error: true
Expand Down Expand Up @@ -301,3 +284,33 @@ jobs:
csv: ./tests/fixtures/batch/*.csv
schema: ./tests/schemas/demo_*.yml
continue-on-error: true


reports:
name: Reports
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: 8.3
coverage: xdebug
extensions: ast

- name: Install project
run: make build --no-print-directory

- name: 📝 Build Reports
run: make report-all --no-print-directory

- name: Upload Artifacts
uses: actions/upload-artifact@v4
continue-on-error: true
with:
name: Reports
path: build/
44 changes: 22 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
[![Static Badge](https://img.shields.io/badge/Rules-118-green?label=Cell%20rules&labelColor=blue&color=gray)](src/Rules/Cell)
[![Static Badge](https://img.shields.io/badge/Rules-206-green?label=Aggregate%20rules&labelColor=blue&color=gray)](src/Rules/Aggregate)
[![Static Badge](https://img.shields.io/badge/Rules-8-green?label=Extra%20checks&labelColor=blue&color=gray)](#extra-checks)
[![Static Badge](https://img.shields.io/badge/Rules-16/54/10-green?label=Plan%20to%20add&labelColor=gray&color=gray)](tests/schemas/todo.yml)
[![Static Badge](https://img.shields.io/badge/Rules-17/11/25-green?label=Plan%20to%20add&labelColor=gray&color=gray)](tests/schemas/todo.yml)
<!-- auto-update:/rules-counter -->

A console utility designed for validating CSV files against a strictly defined schema and validation rules outlined
Expand All @@ -28,26 +28,26 @@ specifications, making it invaluable in scenarios where data quality and consist
- [Introduction](#introduction)
- [Why?](#why)
- [Features](#features)
- [Live Demo](#live-demo)
- [Live demo](#live-demo)
- [Usage](#usage)
- [GitHub Action](#github-action)
- [Docker container](#docker-container)
- [PHP binary](#php-binary)
- [Schema Definition](#schema-definition)
- [Phar binary](#phar-binary)
- [Schema definition](#schema-definition)
- [Example Schema in YAML](#example-schema-in-yaml)
- [Full Schema Description](#full-schema-description)
- [Extra Checks](#extra-checks)
- [Complete CLI Help Message](#complete-cli-help-message)
- [Report Examples](#report-examples)
- [Full schema description](#full-schema-description)
- [Extra checks](#extra-checks)
- [Complete CLI help message](#complete-cli-help-message)
- [Report examples](#report-examples)
- [Benchmarks](#benchmarks)
- [Brief Conclusions](#brief-conclusions)
- [Brief conclusions](#brief-conclusions)
- [Examples of CSV Files](#examples-of-csv-files)
- [Run benchmark locally](#run-benchmark-locally)
- [Disadvantages?](#disadvantages)
- [Coming soon](#coming-soon)
- [Contributing](#contributing)
- [License](#license)
- [See Also](#see-also)
- [See also](#see-also)
<!-- auto-update:/toc -->


Expand Down Expand Up @@ -111,7 +111,7 @@ specifications, making it invaluable in scenarios where data quality and consist
</details>


### Live Demo
### Live demo

As a live demonstration of how the tool works, you can explore the super minimal repository
at [JBZoo/Csv-Blueprint-Demo](https://github.com/JBZoo/Csv-Blueprint-Demo). You're encouraged to fork it and experiment
Expand Down Expand Up @@ -203,7 +203,7 @@ cd csv-blueprint
make docker-build # local tag is "jbzoo/csv-blueprint:local"
```

### PHP binary
### Phar binary

<details>
<summary>Click to see using PHAR file</summary>
Expand Down Expand Up @@ -236,7 +236,7 @@ make build

</details>

## Schema Definition
## Schema definition

Define your CSV validation schema in YAML for clear and structured configuration. Alternative formats are also
supported: [JSON](schema-examples/full.json) and [PHP](schema-examples/full.php), accommodating various preferences and
Expand Down Expand Up @@ -273,7 +273,7 @@ columns:
<!-- auto-update:/readme-sample-yml -->


### Full Schema Description
### Full schema description

In the [example YAML file](schema-examples/full.yml), a detailed description of all features is provided. This
documentation is verified through automated tests, ensuring it remains current.
Expand Down Expand Up @@ -898,7 +898,7 @@ columns:
<!-- auto-update:/full-yml -->


### Extra Checks
### Extra checks

In addition to the schema outlined in the YAML configuration, the tool performs several underlying checks by default to
ensure thorough validation of your CSV files.
Expand All @@ -917,7 +917,7 @@ ensure thorough validation of your CSV files.
These additional checks further secure the integrity and consistency of your CSV data against the defined validation schema.


## Complete CLI Help Message
## Complete CLI help message

This section outlines all available options and commands provided by the tool, leveraging the JBZoo/Cli package for its
CLI. The tool offers a comprehensive set of options to cater to various needs and scenarios, ensuring flexibility and
Expand All @@ -933,7 +933,7 @@ application of the CLI commands, helping users make the most out of the tool's c
<summary>CLICK to see validate:csv help messege</summary>

<!-- auto-update:validate-csv-help -->
```text
```
Description:
Validate CSV file(s) by schema(s).
Expand Down Expand Up @@ -995,7 +995,7 @@ Options:
<summary>CLICK to see validate:schema help messege</summary>

<!-- auto-update:validate-schema-help -->
```text
```
Description:
Validate syntax in schema file(s).
Expand Down Expand Up @@ -1043,7 +1043,7 @@ Options:

</details>

## Report Examples
## Report examples

The validation process culminates in a human-readable report detailing any errors identified within the CSV file. While
the default report format is a table, the tool supports various output formats, including text, GitHub, GitLab,
Expand Down Expand Up @@ -1221,9 +1221,9 @@ devices.

Conversely, tests conducted on a **MacBook Pro (2019) with an Intel 2.4 GHz processor** align closely with the GitHub
Actions results, suggesting that the benchmark table provided reflects an average performance level for typical
engineering hardware. However, it's important to understan
engineering hardware.

### Brief Conclusions
### Brief conclusions

* **Cell Rules**: These rules are highly CPU-intensive but require minimal RAM, typically around 1-2 MB at peak. The
more cell rules applied to a column, the longer the validation process takes due to the additional actions performed
Expand Down Expand Up @@ -1441,7 +1441,7 @@ make codestyle
[MIT License](LICENSE): It's like free pizza - enjoy it, share it, just don't sell it as your own. And remember, no warranty for stomach aches! 😅


## See Also
## See also

- [Cli](https://github.com/JBZoo/Cli) - Framework helps create complex CLI apps and provides new tools for Symfony/Console.
- [CI-Report-Converter](https://github.com/JBZoo/CI-Report-Converter) - It converts different error reporting standards for popular CI systems.
Expand Down

0 comments on commit a8087c2

Please sign in to comment.