Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
7fd7d6d
Migrate files
CBroz1 Feb 22, 2023
caa2823
Make workflow compatible with latest version of the element
kushalbakshi Apr 13, 2023
e698194
Updates for changes in the corresponding element
kushalbakshi Apr 18, 2023
fb3d261
Initial tutorial notebook created
kushalbakshi Apr 20, 2023
9c1eea2
Add pytest and devcontainer files
kushalbakshi Apr 20, 2023
b3be149
General code cleanup and update notebooks
kushalbakshi Apr 27, 2023
33093b7
Add CICD, update CHANGELOG + version
kushalbakshi Apr 27, 2023
eb01692
Update CHANGELOG.md
kushalbakshi Apr 27, 2023
8ce8660
Update .devcontainer/Dockerfile
kushalbakshi Apr 27, 2023
90044d3
Update .github/ISSUE_TEMPLATE/config.yml
kushalbakshi Apr 27, 2023
59c6f63
Update tests/test_pipeline_generation.py
kushalbakshi Apr 27, 2023
1ef1edd
Apply suggestions from code review
kushalbakshi Apr 27, 2023
d39e3b4
Update CICD folder, version-pin requirements
kushalbakshi Apr 27, 2023
92d6dae
Delete .csv files
kushalbakshi Apr 27, 2023
8fc3b7f
Update tests/test_pipeline_generation.py
kushalbakshi Apr 27, 2023
6a0cc31
Remove `Docker` directory + update devcontainer.json
kushalbakshi Apr 27, 2023
1ad1f3e
Merge branch 'main' of https://github.com/kushalbakshi/workflow-zstack
kushalbakshi Apr 27, 2023
7cdba76
Update .devcontainer/devcontainer.json
kushalbakshi Apr 27, 2023
3f53628
Update .devcontainer/devcontainer.json
kushalbakshi Apr 27, 2023
43cd684
Update .devcontainer/devcontainer.json
kushalbakshi Apr 27, 2023
66a2a7a
Update .devcontainer/local/devcontainer.json
kushalbakshi May 3, 2023
833e5be
Rename local to local-test
kushalbakshi May 3, 2023
368dee5
Merge branch 'main' of https://github.com/kushalbakshi/workflow-zstack
kushalbakshi May 3, 2023
db789d4
Update gitignore
kabilar May 5, 2023
67cf768
Add Compose file
kabilar May 6, 2023
73d78ef
Update setup.py
kabilar May 6, 2023
d1bb2bc
Update Dev Container
kabilar May 6, 2023
e2462d4
Update readme
kabilar May 6, 2023
cd10451
Update notebook
kabilar May 6, 2023
a492630
Update Dev Container
kabilar May 10, 2023
836f40a
Update dev container
kabilar May 10, 2023
27d311c
Update requirement
kabilar May 10, 2023
adcfa2f
Update requirements
kabilar May 10, 2023
4d2ce8c
Update readme
kabilar May 11, 2023
0ac759e
Merge branch 'main' of https://github.com/kabilar/workflow-zstack int…
kabilar May 11, 2023
aabbbf0
Merge pull request #1 from kabilar/main
kushalbakshi May 13, 2023
55dd878
Update notebook + workflow
kushalbakshi May 13, 2023
7e75c34
Update notebook + tests
kushalbakshi May 13, 2023
3ae484e
Apply suggestions from code review
kushalbakshi May 15, 2023
2836ab6
Remove `dj_config` from pytests
kushalbakshi May 15, 2023
2ebd444
Merge branch 'main' of https://github.com/kushalbakshi/workflow-zstack
kushalbakshi May 15, 2023
758a116
Apply suggestions from code review
kushalbakshi May 15, 2023
5fe35dc
Add BossDB API info to `test_export`
kushalbakshi May 15, 2023
7f99563
Merge branch 'main' of https://github.com/kushalbakshi/workflow-zstack
kushalbakshi May 15, 2023
fb43b49
Apply suggestions from code review
kushalbakshi May 16, 2023
c9c44c6
Remove old functions from conftest.py
kushalbakshi May 16, 2023
ae643b6
Update requirements
kabilar May 16, 2023
5f9ec99
Update requirements for Dev Container
kushalbakshi May 16, 2023
adae8bd
Update S3 location
kushalbakshi May 16, 2023
8e54e59
Update __init__.py to get volume_root_data_dir
kushalbakshi May 16, 2023
ab84420
Update `paths.py` to work with Codespaces
kushalbakshi May 16, 2023
41beccf
Update `paths.py` and `pipeline.py` to fix workflow errors
kushalbakshi May 16, 2023
9d89473
Update workflow_zstack/paths.py
kushalbakshi May 16, 2023
cda192f
Lost transaction error: Attempted fix 1
kushalbakshi May 16, 2023
f3dbadd
Merge branch 'main' of https://github.com/kushalbakshi/workflow-zstack
kushalbakshi May 16, 2023
e8368c4
Update session directory in notebook
kushalbakshi May 16, 2023
b35d7cb
Apply suggestions from code review
kushalbakshi May 16, 2023
aeec2f8
Update tests/conftest.py
kushalbakshi May 16, 2023
4796a9d
Add assertion when returning tif files
kushalbakshi May 17, 2023
eb3bd43
Update pytest options
kushalbakshi May 17, 2023
daa15ef
Merge 'main' of kushalbakshi/workflow-zstack
kabilar May 17, 2023
cf85d23
Update requirements
kabilar May 17, 2023
6c1c08a
Update workflow_zstack/paths.py
kushalbakshi May 17, 2023
e4f1e4e
Merge branch 'main' of https://github.com/kushalbakshi/workflow-zstack
kushalbakshi May 17, 2023
90ecb34
Update pytests
kushalbakshi May 17, 2023
74330f9
Remove dev container files and update README
kushalbakshi May 18, 2023
41c5a39
Revert "Remove dev container files and update README"
kushalbakshi May 18, 2023
f74df61
Update requirements.txt
kushalbakshi May 19, 2023
84c1ee1
Remove dev container + volume_matching, update notebook + README
kushalbakshi May 19, 2023
c929fb9
Merge branch 'main' of https://github.com/kushalbakshi/workflow-zstack
kushalbakshi May 19, 2023
521f63c
Update tests/test_export.py
kushalbakshi May 19, 2023
ad4e206
Remove volume_matching from tests
kushalbakshi May 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions .cspell.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
// cSpell Settings
//https://github.com/streetsidesoftware/vscode-spell-checker
{
"version": "0.2", // Version of the setting file. Always 0.2
"language": "en", // language - current active spelling language
"enabledLanguageIds": [
"markdown",
"yaml",
"python"
],
// flagWords - list of words to be always considered incorrect
// This is useful for offensive words and common spelling errors.
// For example "hte" should be "the"
"flagWords": [],
"allowCompoundWords": true,
"ignorePaths": [
"./element_*.egg-info/*",
"./images/*"
],
"words": [
"asarray",
"astype",
"Berens",
"bossdb",
"CICD",
"connectome",
"Connectomics",
"DBURLs",
"djarchive",
"DJARCHIVE",
"Ecker",
"elif",
"Ephys",
"genotyping",
"Hoenselaar",
"IACUC",
"inlinehilite",
"Kasthuri",
"linenums",
"mkdocs",
"mkdocstrings",
"numpy",
"pymdownx",
"pyproject",
"pytest",
"Reimer",
"Roboto",
"RRID",
"Rxiv",
"Sasaki",
"segmentations",
"Shen",
"Siapas",
"Sinz",
"Sitonic",
"Tolias",
"voxel",
"witvliet",
"Yatsenko",
"Zuckerman"
]
}
39 changes: 39 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: 'bug'
assignees: ''

---

## Bug Report

### Description

A clear and concise description of what is the overall operation that is intended to be
performed that resulted in an error.

### Reproducibility
Include:
- OS (WIN | MACOS | Linux)
- DataJoint Element Version
- MySQL Version
- MySQL Deployment Strategy (local-native | local-docker | remote)
- Minimum number of steps to reliably reproduce the issue
- Complete error stack as a result of evaluating the above steps

### Expected Behavior
A clear and concise description of what you expected to happen.

### Screenshots
If applicable, add screenshots to help explain your problem.

### Additional Research and Context
Add any additional research or context that was conducted in creating this report.

For example:
- Related GitHub issues and PR's either within this repository or in other relevant
repositories.
- Specific links to specific lines or a focus within source code.
- Relevant summary of Maintainers development meetings, milestones, projects, etc.
5 changes: 5 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: DataJoint Contribution Guideline
url: https://datajoint.com/docs/community/contribute/
about: Please make sure to review the DataJoint Contribution Guidelines
57 changes: 57 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
name: Feature request
about: Suggest an idea for a new feature
title: ''
labels: 'enhancement'
assignees: ''

---

## Feature Request

### Problem

A clear and concise description how this idea has manifested and the context. Elaborate
on the need for this feature and/or what could be improved. Ex. I'm always frustrated
when [...]

### Requirements

A clear and concise description of the requirements to satisfy the new feature. Detail
what you expect from a successful implementation of the feature. Ex. When using this
feature, it should [...]

### Justification

Provide the key benefits in making this a supported feature. Ex. Adding support for this
feature would ensure [...]

### Alternative Considerations

Do you currently have a work-around for this? Provide any alternative solutions or
features you've considered.

### Related Errors
Add any errors as a direct result of not exposing this feature.

Please include steps to reproduce provided errors as follows:
- OS (WIN | MACOS | Linux)
- DataJoint Element Version
- MySQL Version
- MySQL Deployment Strategy (local-native | local-docker | remote)
- Minimum number of steps to reliably reproduce the issue
- Complete error stack as a result of evaluating the above steps

### Screenshots
If applicable, add screenshots to help explain your feature.

### Additional Research and Context
Add any additional research or context that was conducted in creating this feature request.

For example:
- Related GitHub issues and PR's either within this repository or in other relevant
repositories.
- Specific links to specific lines or a focus within source code.
- Relevant summary of Maintainers development meetings, milestones, projects, etc.
- Any additional supplemental web references or links that would further justify this
feature request.
18 changes: 18 additions & 0 deletions .github/workflows/u24_workflow_before_release.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: u24_workflow_before_release_0.0.1
on:
pull_request:
push:
branches:
- '**'
tags-ignore:
- '**'
workflow_dispatch:
jobs:
call_context_check:
uses: dj-sciops/djsciops-cicd/.github/workflows/context_check.yaml@main
call_u24_workflow_build_debian:
uses: dj-sciops/djsciops-cicd/.github/workflows/u24_workflow_build.yaml@main
with:
jhub_ver: 1.4.2
py_ver: 3.9
dist: debian
20 changes: 20 additions & 0 deletions .github/workflows/u24_workflow_release_call.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: u24_workflow_release_call_0.0.1
on:
workflow_run:
workflows: ["u24_workflow_tag_to_release_0.0.1"]
types:
- completed
jobs:
call_context_check:
uses: dj-sciops/djsciops-cicd/.github/workflows/context_check.yaml@main
call_u24_workflow_release_debian:
if: >-
github.event.workflow_run.conclusion == 'success' && github.repository_owner == 'datajoint'
uses: dj-sciops/djsciops-cicd/.github/workflows/u24_workflow_release.yaml@main
with:
jhub_ver: 1.4.2
py_ver: 3.9
dist: debian
secrets:
REGISTRY_USERNAME: ${{secrets.DOCKER_USERNAME}}
REGISTRY_PASSWORD: ${{secrets.DOCKER_PASSWORD}}
15 changes: 15 additions & 0 deletions .github/workflows/u24_workflow_tag_to_release.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: u24_workflow_tag_to_release_0.0.1
on:
push:
tags:
- '*.*.*'
- 'test*.*.*'
jobs:
call_context_check:
uses: dj-sciops/djsciops-cicd/.github/workflows/context_check.yaml@main
call_u24_workflow_build_debian:
uses: dj-sciops/djsciops-cicd/.github/workflows/u24_workflow_build.yaml@main
with:
jhub_ver: 1.4.2
py_ver: 3.9
dist: debian
80 changes: 80 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# User files and data directories
example_data/
test.ipynb

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# Distribution, packaging, PyInstaller
.Python
env/
build/
*egg*/
*dist/
downloads/
lib*/
parts/
var/
wheels/
.installed.cfg
*.egg
*.manifest
*.spec
pip-*.txt

# Unit test / coverage reports
htmlcov/
.tox/
*.cov*
.cache
nosetests.xml
coverage.xml
.hypothesis/
.pytest_cache/

# C extension, Translations
# editors: vscode, emacs, Mac
*.so
*.mo
*.pot
.vscode
**/*~
**/#*#
**/.#*
.DS_Store

# Django, Flask, Scrapy, Sphinx, mkdocs
# PyBuilder, Jupyter, SageMath, celery beat
*.log
local_settings.py
instance/
.webassets-cache
.scrapy
scratchpaper.*
docs/_build/
/site
target/
.*checkpoints
celerybeat-schedule
*.sage.py

# dotenv, virtualenv, pyenv, mypy
.*env
venv/
ENV/
.python-version
.mypy_cache/

# Spyder/Rope project settings
.spy*project
.ropeproject

# datajoint, notes, nwb export
dj_local_c*.json
temp*
*nwb

# vscode
*.code-workspace
58 changes: 58 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
default_stages: [commit, push]
exclude: (^.github/|^docs/|^images/|^notebooks/py_scripts/)

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files # prevent giant files from being committed
- id: requirements-txt-fixer
- id: mixed-line-ending
args: ["--fix=lf"]
description: Forces to replace line ending by the UNIX 'lf' character.

# black
- repo: https://github.com/psf/black
rev: 22.12.0
hooks:
- id: black
- id: black-jupyter
args:
- --line-length=88

# isort
- repo: https://github.com/pycqa/isort
rev: 5.12.0
hooks:
- id: isort
args: ["--profile", "black"]
description: Sorts imports in an alphabetical order

# flake8
- repo: https://github.com/pycqa/flake8
rev: 4.0.1
hooks:
- id: flake8
args: # arguments to configure flake8
# making isort line length compatible with black
- "--max-line-length=88"
- "--max-complexity=18"
- "--select=B,C,E,F,W,T4,B9"

# these are errors that will be ignored by flake8
# https://www.flake8rules.com/rules/{code}.html
- "--ignore=E203,E501,W503,W605,E402"
# E203 - Colons should not have any space before them.
# Needed for list indexing
# E501 - Line lengths are recommended to be no greater than 79 characters.
# Needed as we conform to 88
# W503 - Line breaks should occur after the binary operator.
# Needed because not compatible with black
# W605 - a backslash-character pair that is not a valid escape sequence now
# generates a DeprecationWarning. This will eventually become a SyntaxError.
# Needed because we use \d as an escape sequence
# E402 - Place module level import at the top.
# Needed to prevent circular import error
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Changelog

Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.

## [0.1.0] - Unreleased

+ Add - Workflow pipeline, pytests, CICD

[0.1.0]: https://github.com/datajoint/workflow-zstack/releases/tag/0.1.0
Loading