Skip to content
This repository has been archived by the owner on Apr 12, 2020. It is now read-only.

Commit

Permalink
approve changes
Browse files Browse the repository at this point in the history
  • Loading branch information
Nick.Vogt committed Feb 19, 2019
2 parents a6d2afe + 18b7605 commit 8cdb24e
Show file tree
Hide file tree
Showing 8 changed files with 230 additions and 29 deletions.
23 changes: 23 additions & 0 deletions .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[bumpversion]
current_version = 0.4.0
commit = True
tag = False
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)
(\.(?P<release>[a-z]+)(?P<build>\d+))?
serialize =
{major}.{minor}.{patch}.{release}{build}
{major}.{minor}.{patch}

[bumpversion:part:release]
optional_value = placeholder
first_value = dev
values =
dev
placeholder

[bumpversion:part:build]

[bumpversion:file:./dequindre/__init__.py]

59 changes: 59 additions & 0 deletions contributing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Contributing

When contributing to this repository, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository before making a change.

Please note we have a code of conduct, please follow it in all your interactions with the project.

## Pull Request Process

1. Ensure any install or build dependencies are removed before the end of the layer when doing a build.
2. Update the README.md with details of changes to the interface, this includes new environment variables, exposed ports, useful file locations and container parameters.
3. Increase the version numbers in any examples files and the README.md to the new version that this Pull Request would represent. The versioning scheme we use is [SemVer](http://semver.org/).
4. You may merge the Pull Request in once you have the sign-off of the author.

## Code of Conduct

### Our Pledge

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.

### Our Standards

Examples of behavior that contributes to creating a positive environment include:

* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

* The use of sexualized language or imagery and unwelcome sexual attention or advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a professional setting

### Our Responsibilities

Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.

### Scope

This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.

### Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the author at vogt4nick@gmail.com. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.

### Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]

[homepage]: http://contributor-covenant.org
[version]: http://contributor-covenant.org/version/1/4/
11 changes: 7 additions & 4 deletions dequindre/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
from time import sleep


__version__ = '0.4.0'


class CyclicGraphError(Exception):
pass

Expand Down Expand Up @@ -52,8 +55,7 @@ class Task:
loc (str): location of the python script that runs the task.
env (str): Which environment to run.
"""
def __init__(self, loc: str, env: str):
# check_conda()
def __init__(self, loc: str, env: str = 'base'):
assert isinstance(loc, str), 'loc must be a str'
assert len(loc) > 0, 'loc cannot be an empty string'
assert isinstance(env, str), 'env must be a str'
Expand Down Expand Up @@ -338,8 +340,9 @@ class Dequindre:
TODO: The DAG should catch cycles before they get to Dequindre.
TODO: Define exception for when cycles are detected
"""
def __init__(self, dag: DAG):
check_conda()
def __init__(self, dag: DAG, *, validate_conda: bool = True):
if validate_conda:
check_conda()
self.original_dag = dag
self.refresh_dag()

Expand Down
15 changes: 6 additions & 9 deletions dequindre/tests/test_Task.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,9 @@ def test__Task_init():
good_loc = 'path/to/file.py'
good_env = 'test-env'

assert isinstance(
Task(loc=good_loc, env=good_env),
Task
)
with pytest.raises(TypeError):
Task()

with pytest.raises(AssertionError):
Task(loc=None, env=good_env)

Expand All @@ -26,17 +25,15 @@ def test__Task_init():
with pytest.raises(AssertionError):
Task(loc=good_loc, env='')

Task(loc=good_loc)
Task(loc=good_loc, env=good_env)


def test__Task_repr():
make_tea = Task('make_tea.py', 'test-env')
assert repr(make_tea) == "Task(make_tea.py)"


def test__Task_repr():
make_tea = Task('make_tea', 1, 'test-env')
assert repr(make_tea) == "Task(loc=make_tea, stage=1, env=test-env)"


def test__Task_hash():
A = Task('test.py', 'test-env')
B = Task('test.py', 'test-env')
Expand Down
13 changes: 13 additions & 0 deletions environment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: dequindre-dev
channels:
- conda-forge
- defaults
dependencies:
- python =3.6
- pip >=18.1
- pytest >=4.2, < 5.0
- sphinx >=1.8, < 2.0
- twine >=1.13, < 2.0
- pip:
- bump2version >=0.5.2, <0.6
- readme-renderer
4 changes: 2 additions & 2 deletions license.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Copyright (c) 2011-2017 GitHub Inc.
Copyright (c) 2019 dequindre

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
110 changes: 102 additions & 8 deletions readme.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,105 @@

```dequindre-logo
_ _ _
__| | ___ __ _ _ _(_)_ __ __| |_ __ ___
/ _` |/ _ \/ _` | | | | | '_ \ / _` | '__/ _ \
| (_| | __/ (_| | |_| | | | | | (_| | | | __/
\__,_|\___|\__, |\__,_|_|_| |_|\__,_|_| \___|
|_|
# Dequindre /_de-KWIN-der_/ (n.): A minimalist scheduler.

Dequindre is built for professionals with the most basic use-cases in mind. You can install and configure your first pipeline in minutes. Dequindre makes it easy to configure, test, and deploy workflows. It also functions as a learning tool for students and professionals without the time or resources to setup the requisite architecture for a full-featured scheduler.

Inspired by the complexity of Airflow, Dequindre is build on three design pillars:

1. The schedule should be human readable and optimizable.
2. Tasks should fail loudly and clearly.
3. 80% of the work can be done with the bare-essential features.

Dequindre offers

* No Python dependencies; no third-party bugs
* Single-container deployment
* Legible source code; fewer than 1000 lines
* Fast, dynamic configuration

# Usage

Dequindre allows dynamic configuration with Python. By example, we may program the process to make tea as

```usage
>>> from dequindre import Task, DAG, Dequindre
>>>
>>> # define tasks and environments
>>> pour_water = Task('pour_water.py, env='kitchen')
>>> boil_water = Task('boil_water.py', env='microwave')
>>> prep_infuser = Task('prep_infuser.py', env='kitchen')
>>> steep_tea = Task('steep_tea.py', env='kitchen')
>>>
>>> # define runtime dependencies
>>> make_tea = DAG(dependencies={
... boil_water: {pour_water},
... steep_tea: {boil_water, prep_infuser}
...})
>>>
>>> # run tasks
>>> dq = Dequindre(dag, check_conda=False)
>>> dq = dq.get_priorities()
defaultdict(<class 'set'>, {
1: {Task(make_tea.py), Task(prep_infuser.py)},
2: {Task(boil_water.py)},
3: {Task(steep_tea.py)}
})
>>> dq.run_tasks()
...
```

`dq.run_tasks()` looks at the priorities and runs `conda run -n ENVIRONMENT python PATH` under the hood.

## Getting Started

Dequindre has two requirements: conda and python.

### Prerequisites

Dequindre relies on [conda >=4.6](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) to run tasks in different environments.

```conda-version
$ conda --version
conda 4.6.4
```

### Installing

```pip
$ pip install dequindre
```

## Contributing

If you're interested in contributing to Dequindre, [raise an issue](https://github.com/vogt4nick/dequindre/issues), make a pull request to `dev`, and reach out to the author, vogt4nick.

Please read [contributing.md](contributing.md) for details on our code of conduct, and the process for submitting pull requests to us.

### Running the tests

Dequindre's tests are currently written entirely with the `pytest` module. Navigate to the repo's directory on your system and use the `pytest` command. Output should resemble

```pytest
$ pytest
============================= test session starts =============================
platform win32 -- Python 3.6.6, pytest-4.2.1, py-1.7.0, pluggy-0.8.1
rootdir: C:\Users\Nick.Vogt\Projects\repos\dequindre, inifile:
collected 25 items
dequindre\tests\test_DAG.py ............... [ 60%]
dequindre\tests\test_Dequindre.py ...... [ 84%]
dequindre\tests\test_Task.py .... [100%]
========================== 25 passed in 4.14 seconds ==========================
```

Dequindre (_deh-KWIN-der_) is a minimalist scheduler for workflow automation. Dequindre makes it easy to configure, test, and deploy POC data pipelines. It also functions as a learning tool for students and professionals without the time or resources to setup the requisite architecture for a production data pipeline.
## Versioning

We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://github.com/vogt4nick/dequindre/tags).

## License

This project is licensed under the MIT License - see the [license.md](license.md) file for details.

## Acknowledgements

Thank you, Dynatrace, for facilitating the early development of Dequindre during Innovation Day, February 2019.
24 changes: 18 additions & 6 deletions setup.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,28 @@
from setuptools import setup, find_packages


version = {}
with open('./dequindre/__init__.py', 'r') as ifile:
exec(ifile.read(), version)

setup(
name='dequindre',
version='0.3.0',
description="Dequindre: a lightweight scheduler.",
version=version['__version__'],
description=(
"Dequindre /_de-KWIN-der_/ (n.): "
"A minimalist scheduler for simple workflow automation."
),
long_description=open('readme.md').read(),
long_description_content_type="text/markdown",
url='https://github.com/vogt4nick/dequindre',
author='Nick Vogt',
author_email='vogt4nick@gmail.com',
license='MIT',
packages=["dequindre"],
# install_requires='',
test_suite='nose.collector',
tests_require=['nose']
classifiers=[
"Development Status :: 2 - Pre-Alpha",
"Programming Language :: Python",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent"
],
include_package_data=True
)

0 comments on commit 8cdb24e

Please sign in to comment.