Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure workflows steps #343

Merged
merged 150 commits into from
May 7, 2024
Merged
Show file tree
Hide file tree
Changes from 149 commits
Commits
Show all changes
150 commits
Select commit Hold shift + click to select a range
665bd17
Add foundations for change in steps definitions
PawelPeczek-Roboflow Apr 3, 2024
ce8fd3a
Merge branch 'main' into feature/restructure_workflows_steps
PawelPeczek-Roboflow Apr 3, 2024
f499215
Refactor Roboflow classifier step
PawelPeczek-Roboflow Apr 3, 2024
458918a
Add multi-label classification
PawelPeczek-Roboflow Apr 3, 2024
ffb483d
Transform roboflow models blocks to new schema of blocks
PawelPeczek-Roboflow Apr 3, 2024
5721259
Add foundations modules for blocks
PawelPeczek-Roboflow Apr 3, 2024
b465e4d
Port majority of blocks
PawelPeczek-Roboflow Apr 4, 2024
f1e661a
Port additional blocks
PawelPeczek-Roboflow Apr 4, 2024
696a634
Port LMM steps
PawelPeczek-Roboflow Apr 5, 2024
6e455f3
Provide a logic to implement initialisation of steps
PawelPeczek-Roboflow Apr 8, 2024
28dead6
WIP - part of work reuqired to construct and validate execution graph
PawelPeczek-Roboflow Apr 9, 2024
fdae1a3
Finalise first draft of new compiler
PawelPeczek-Roboflow Apr 9, 2024
1d4c0ae
Start building executor
PawelPeczek-Roboflow Apr 9, 2024
1a2b04f
Add module to keep step outputs cached
PawelPeczek-Roboflow Apr 10, 2024
81a8bbd
Complete first draft of new execution engine
PawelPeczek-Roboflow Apr 10, 2024
eda2a61
First rooughly working executor
PawelPeczek-Roboflow Apr 10, 2024
6c00172
Fix issue with windows listing files twice
PawelPeczek-Roboflow Apr 12, 2024
b7404e4
Add error handling for parsing and graph construction
PawelPeczek-Roboflow Apr 12, 2024
d145883
Add better error handling for new workflows execution engine
PawelPeczek-Roboflow Apr 12, 2024
f34291b
Provide demo for workflow execution engine in InferencePipeline
PawelPeczek-Roboflow Apr 12, 2024
c9db7dd
Kick of the docs
PawelPeczek-Roboflow Apr 15, 2024
92b9f68
Add docs
PawelPeczek-Roboflow Apr 16, 2024
340fdb0
Fix tests
PawelPeczek-Roboflow Apr 16, 2024
467f80a
Make linter happy
PawelPeczek-Roboflow Apr 16, 2024
e1f0420
Fix imports
PawelPeczek-Roboflow Apr 16, 2024
e8e235b
Fix imports
PawelPeczek-Roboflow Apr 16, 2024
564320b
Fix makefile
PawelPeczek-Roboflow Apr 16, 2024
f13e4e6
Make changes to properly distinguish between batch and non-batch
PawelPeczek-Roboflow Apr 17, 2024
c9de256
Fix not imported symbol
PawelPeczek-Roboflow Apr 17, 2024
b879b76
Fix not imported symbol
PawelPeczek-Roboflow Apr 17, 2024
b02ec72
Add batch of boolean
PawelPeczek-Roboflow Apr 17, 2024
da1e5c9
Add batch/singular modes for inputs and outputs
PawelPeczek-Roboflow Apr 17, 2024
fb3be52
initial docs
SkalskiP Apr 17, 2024
73ba5f2
inputs documented
SkalskiP Apr 17, 2024
6e1c5d2
Fix issue with pulling multiple selectors from list
PawelPeczek-Roboflow Apr 18, 2024
78f52cd
with love for James :*
SkalskiP Apr 18, 2024
25d4108
Fix inference pipeline for workflows and add kind
PawelPeczek-Roboflow Apr 18, 2024
9443936
Fix naming convention
PawelPeczek-Roboflow Apr 18, 2024
3cb25a0
long and short descriptions added
SkalskiP Apr 18, 2024
905b4d4
Merge fresh origin branch
PawelPeczek-Roboflow Apr 18, 2024
278b0a5
Make linters happy
PawelPeczek-Roboflow Apr 18, 2024
4e01bc8
Add generated docs to gitignore
PawelPeczek-Roboflow Apr 18, 2024
775f6bf
Get rid of uncercognised types
PawelPeczek-Roboflow Apr 18, 2024
8abd0ba
licenses added
SkalskiP Apr 18, 2024
f37ee23
available connections added
SkalskiP Apr 18, 2024
f31c095
Merge
PawelPeczek-Roboflow Apr 18, 2024
2c5f5e6
Fix table
PawelPeczek-Roboflow Apr 18, 2024
92cf515
format available blocks with links
SkalskiP Apr 18, 2024
6b3bd69
Fix bug with list of ref
PawelPeczek-Roboflow Apr 18, 2024
f3747be
refs styles
SkalskiP Apr 18, 2024
e8486b6
bindings docs refactor
SkalskiP Apr 18, 2024
181ed18
small fixes
SkalskiP Apr 18, 2024
884fd6c
Run black against docs building script
PawelPeczek-Roboflow Apr 22, 2024
96aa6ca
Fix couple of issues
PawelPeczek-Roboflow Apr 22, 2024
c9470b2
Merge pull request #360 from roboflow/feature/restructure_workflows_s…
PawelPeczek-Roboflow Apr 22, 2024
aee7f47
Work in progress - build introspection components
PawelPeczek-Roboflow Apr 23, 2024
3a7c2f2
Add changes to make it possible to generate docs about klinds
PawelPeczek-Roboflow Apr 24, 2024
9413479
Add JSON example to blocks definitions
PawelPeczek-Roboflow Apr 25, 2024
5863f5e
Add JSON example to blocks definitions
PawelPeczek-Roboflow Apr 25, 2024
408bb7b
Fix errors spotted while testing
PawelPeczek-Roboflow Apr 25, 2024
65e2889
Make linters happy
PawelPeczek-Roboflow Apr 25, 2024
6582ef8
Create tmp docs building
PawelPeczek-Roboflow Apr 25, 2024
8025578
Create tmp docs building
PawelPeczek-Roboflow Apr 25, 2024
2838e48
Create tmp docs building
PawelPeczek-Roboflow Apr 25, 2024
aa26c57
Create tmp docs building
PawelPeczek-Roboflow Apr 25, 2024
f691c31
Create tmp docs building
PawelPeczek-Roboflow Apr 25, 2024
b5d23f9
Create tmp docs building
PawelPeczek-Roboflow Apr 25, 2024
f118aa5
Create tmp docs building
PawelPeczek-Roboflow Apr 25, 2024
fa7ca0c
Create tmp docs building
PawelPeczek-Roboflow Apr 25, 2024
051cc5f
Add enterprise workflows
PawelPeczek-Roboflow Apr 25, 2024
113c79f
Add enterprise workflows
PawelPeczek-Roboflow Apr 25, 2024
32e4aaf
Add gh token
PawelPeczek-Roboflow Apr 25, 2024
588f3d2
Add gh token
PawelPeczek-Roboflow Apr 26, 2024
ba7ba39
Get rid of old workflows
PawelPeczek-Roboflow Apr 26, 2024
d2caf3b
Restructure docs
PawelPeczek-Roboflow Apr 26, 2024
68cacf4
Merge with main
PawelPeczek-Roboflow Apr 26, 2024
e192777
Make linters happy
PawelPeczek-Roboflow Apr 26, 2024
5127ff0
Fix AL data collector
PawelPeczek-Roboflow Apr 26, 2024
2bf8b84
Make linters happy
PawelPeczek-Roboflow Apr 26, 2024
620e9bb
Fix circular dependency issue
PawelPeczek-Roboflow Apr 26, 2024
63a642d
Fix imports
PawelPeczek-Roboflow Apr 26, 2024
c44365d
Make linters happy
PawelPeczek-Roboflow Apr 26, 2024
bee8304
Get rid of not needed errors definitions
PawelPeczek-Roboflow Apr 26, 2024
e1f429a
Add unit tests for workflows/introspection/schema_parser
PawelPeczek-Roboflow Apr 26, 2024
2763b22
Add unit tests for connections discovery
PawelPeczek-Roboflow Apr 26, 2024
bc4f542
Add tests to loading
PawelPeczek-Roboflow Apr 26, 2024
ae968ba
Add unit tests for workflows core steps manifests parsing
PawelPeczek-Roboflow Apr 29, 2024
914fc1b
Add unit tests for workflows core steps manifests parsing
PawelPeczek-Roboflow Apr 29, 2024
1c65319
Add tests for graph constructor
PawelPeczek-Roboflow Apr 29, 2024
0de7a9a
Add tests for kinds check
PawelPeczek-Roboflow Apr 29, 2024
db7ef8c
Add tests for steps initialisation
PawelPeczek-Roboflow Apr 29, 2024
6455780
Make linters happy
PawelPeczek-Roboflow Apr 29, 2024
6886a60
Add test to cover inputs substitutions
PawelPeczek-Roboflow Apr 29, 2024
7551017
Add first batch of tests for execution engine:
PawelPeczek-Roboflow Apr 29, 2024
814bcaf
Add tests for outputs constructor
PawelPeczek-Roboflow Apr 30, 2024
194e548
Add remaining unit tests for execution engine
PawelPeczek-Roboflow Apr 30, 2024
ff48a1d
Add changes to SDK errors handler
PawelPeczek-Roboflow Apr 30, 2024
b434cac
Merge branch 'main' into feature/restructure_workflows_steps
PawelPeczek-Roboflow Apr 30, 2024
5f48a12
Fix problem with getting config from roboflow API
PawelPeczek-Roboflow Apr 30, 2024
b9274c8
Fix bug from previous commit
PawelPeczek-Roboflow Apr 30, 2024
acec962
Fix bug with wildcards outputs
PawelPeczek-Roboflow Apr 30, 2024
919bf38
Bump version
PawelPeczek-Roboflow Apr 30, 2024
a133333
Add definition of CI for unit tests of workflows
PawelPeczek-Roboflow May 1, 2024
ff0f39a
Resolve conflicts with main
PawelPeczek-Roboflow May 1, 2024
6db9355
Comment out old tests of workflows
PawelPeczek-Roboflow May 1, 2024
41caf5d
Fix imports
PawelPeczek-Roboflow May 1, 2024
cab8924
Make linters happy
PawelPeczek-Roboflow May 1, 2024
81173c2
Fix tests and definitions of requirements to accomodate for pydantic …
PawelPeczek-Roboflow May 1, 2024
8283fdf
Remove explicit _parent_coordinates output
PawelPeczek-Roboflow May 1, 2024
fb50218
Add first integration tests for workflows
PawelPeczek-Roboflow May 1, 2024
e8e3a23
Point to CPU torch wheels in integration tests install
PawelPeczek-Roboflow May 1, 2024
bb5765f
Add integration tests for static crop-based workflows
PawelPeczek-Roboflow May 1, 2024
d6621c1
Add integration test for multi stage workflow: detection plus ocr
PawelPeczek-Roboflow May 1, 2024
44ce24f
Add requirements to install DocTR model in workflows integration tests
PawelPeczek-Roboflow May 1, 2024
cf90648
Denote in kinds connections if property accepts list of references
PawelPeczek-Roboflow May 1, 2024
d9ad139
Try to fix stability reg. test of detection + crop
PawelPeczek-Roboflow May 2, 2024
6181c58
Add integration tests for detection+classification and flow_controll+…
PawelPeczek-Roboflow May 2, 2024
93826d5
Fix plugin loading such that it can be mocked for older versions of Py
PawelPeczek-Roboflow May 2, 2024
b2c4fef
Apply first CR comments
PawelPeczek-Roboflow May 2, 2024
542b1c1
Apply change issubclass->isinstance
PawelPeczek-Roboflow May 2, 2024
215d902
Apply next portion of CR comments
PawelPeczek-Roboflow May 2, 2024
e240f4a
Apply next portion of CR comments
PawelPeczek-Roboflow May 2, 2024
5af4d26
Make linters happy
PawelPeczek-Roboflow May 2, 2024
438d042
Apply next portion of CR comments
PawelPeczek-Roboflow May 2, 2024
beb5adc
Apply next portion of CR comments
PawelPeczek-Roboflow May 2, 2024
0e80da5
Apply next portion of CR comments
PawelPeczek-Roboflow May 2, 2024
535335f
Change operators
PawelPeczek-Roboflow May 2, 2024
2193d1c
Make changes into naming conventions in entities impacting manifests …
PawelPeczek-Roboflow May 3, 2024
2112296
Revert unwanted changes
PawelPeczek-Roboflow May 3, 2024
8f7530b
Uncomment commented blocks
PawelPeczek-Roboflow May 3, 2024
932a79c
Fix naming conventions in internal functions
PawelPeczek-Roboflow May 3, 2024
951ab4c
Resolve rest of requests for changing names
PawelPeczek-Roboflow May 3, 2024
7607fad
Make linters happy
PawelPeczek-Roboflow May 3, 2024
205c57c
Remove tmp focs building
PawelPeczek-Roboflow May 3, 2024
1da06ad
Bump version
PawelPeczek-Roboflow May 3, 2024
bf7c604
Migrate tests for models
PawelPeczek-Roboflow May 3, 2024
f673989
Add assets
PawelPeczek-Roboflow May 3, 2024
bfea803
Port remaining tests
PawelPeczek-Roboflow May 3, 2024
17db17b
Add extension to export primitive types connections
PawelPeczek-Roboflow May 3, 2024
46ce988
Rename literals for selected elements markers in kinds
PawelPeczek-Roboflow May 3, 2024
44ad1a9
Move methods to detect outputs to manifest entities
PawelPeczek-Roboflow May 3, 2024
1eb2554
Add integration tests for server
PawelPeczek-Roboflow May 3, 2024
e49e1c0
Merge branch 'main' into feature/restructure_workflows_steps
PawelPeczek-Roboflow May 3, 2024
81fd980
Add integration tests for server
PawelPeczek-Roboflow May 3, 2024
b513306
Fix issue with ABC
PawelPeczek-Roboflow May 6, 2024
95c71af
Add workflow dispatch trigger
PawelPeczek-Roboflow May 6, 2024
e6d4c68
Add changes to docs
PawelPeczek-Roboflow May 6, 2024
8a92155
Add fixes for problems detected while testing
PawelPeczek-Roboflow May 6, 2024
942ae86
Bump version
PawelPeczek-Roboflow May 6, 2024
280547e
Update comment about detection offset
PawelPeczek-Roboflow May 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,28 @@ name: Docs Workflow
on:
release:
types: [created]
workflow_dispatch:

jobs:
deploy:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
- uses: actions/checkout@v4
with:
python-version: 3.x
ref: ${{ github.head_ref }}
path: inference_repo
- uses: actions/checkout@v4
with:
repository: roboflow/workflows-enterprise-blocks
ref: main
path: roboflow_enterprise_blocks
token: ${{ secrets.GH_TOKEN_FOR_WORKFLOWS_ENTERPRISE }}
- uses: actions/setup-python@v5
with:
python-version: "3.10"
check-latest: true
- run: pip install mkdocs-material
- run: pip install "mkdocstrings[python]"
- run: pip install "mkdocs-material[imaging]"
Expand All @@ -22,4 +33,15 @@ jobs:
- run: pip install "mkdocs-literate-nav"
- run: pip install "mkdocs-jupyter"
- run: pip install -r requirements/requirements.docs.txt
working-directory: ./inference_repo
- run: |
python -m pip install --upgrade pip
pip install --upgrade setuptools
pip install -r requirements/_requirements.txt -r requirements/requirements.cpu.txt -r requirements/requirements.sdk.http.txt -r requirements/requirements.test.unit.txt -r requirements/requirements.http.txt
working-directory: ./inference_repo
- run: pip install .
working-directory: ./roboflow_enterprise_blocks
- run: python -m development.docs.build_block_docs
working-directory: ./inference_repo
- run: mkdocs gh-deploy --force
working-directory: ./inference_repo
2 changes: 1 addition & 1 deletion .github/workflows/integration_tests_inference_models.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install --upgrade setuptools
pip install -r requirements/_requirements.txt -r requirements/requirements.cpu.txt -r requirements/requirements.sdk.http.txt -r requirements/requirements.test.unit.txt -r requirements/requirements.http.txt -r requirements/requirements.yolo_world.txt
pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements/_requirements.txt -r requirements/requirements.cpu.txt -r requirements/requirements.sdk.http.txt -r requirements/requirements.test.unit.txt -r requirements/requirements.http.txt -r requirements/requirements.yolo_world.txt
- name: 🧪 Integration Tests of Inference models
timeout-minutes: 45
run: MAX_BATCH_SIZE=6 python -m pytest tests/inference/models_predictions_tests
30 changes: 30 additions & 0 deletions .github/workflows/integration_tests_workflows_x86.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: INTEGRATION TESTS - workflows

on:
pull_request:
branches: [main]

jobs:
build-dev-test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
steps:
- name: 🛎️ Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.head_ref }}
- name: 🐍 Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
check-latest: true
- name: 📦 Install dependencies
run: |
python -m pip install --upgrade pip
pip install --upgrade setuptools
pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements/_requirements.txt -r requirements/requirements.cpu.txt -r requirements/requirements.sdk.http.txt -r requirements/requirements.test.unit.txt -r requirements/requirements.http.txt -r requirements/requirements.yolo_world.txt -r requirements/requirements.doctr.txt
- name: 🧪 Integration Tests of Workflows
timeout-minutes: 30
run: ROBOFLOW_API_KEY=${{ secrets.API_KEY }} python -m pytest tests/workflows/integration_tests
30 changes: 30 additions & 0 deletions .github/workflows/unit_tests_workflows_x86.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: UNIT TESTS - workflows

on:
pull_request:
branches: [main]

jobs:
build-dev-test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
steps:
- name: 🛎️ Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.head_ref }}
- name: 🐍 Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
check-latest: true
- name: 📦 Install dependencies
run: |
python -m pip install --upgrade pip
pip install --upgrade setuptools
pip install -r requirements/_requirements.txt -r requirements/requirements.cpu.txt -r requirements/requirements.sdk.http.txt -r requirements/requirements.test.unit.txt -r requirements/requirements.http.txt
- name: 🧪 Unit Tests of Workflows
timeout-minutes: 30
run: python -m pytest tests/workflows/unit_tests
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -162,4 +162,8 @@ tests/inference/models_predictions_tests/assets/*.zip
dev.py
scratch
!tests/inference/unit_tests/core/utils/assets/*.png
!tests/inference/unit_tests/core/utils/assets/*.jpg
!tests/inference/unit_tests/core/utils/assets/*.jpg
docs/workflows/blocks/*
docs/workflows/kinds/*
!tests/workflows/integration_tests/assets/*.jpg
!tests/workflows/unit_tests/core_steps/models/third_party/assets/*.png
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ stop_test_docker:

create_wheels:
python -m pip install --upgrade pip
python -m pip install wheel twine requests -r requirements/_requirements.txt -r requirements/requirements.cpu.txt -r requirements/requirements.http.txt
python -m pip install wheel twine requests -r requirements/_requirements.txt -r requirements/requirements.cpu.txt -r requirements/requirements.http.txt -r requirements/requirements.sdk.http.txt
rm -f dist/*
rm -rf build/*
python .release/pypi/inference.core.setup.py bdist_wheel
Expand Down
Loading
Loading