Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data loader #85

Merged
merged 29 commits into from
May 14, 2020
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
a2ae697
reorg utils and start class
MekWarrior Apr 22, 2020
99e287e
Merge branch 'master' into data_loader
MekWarrior Apr 29, 2020
fefcef5
fix init setup throughout repo
MekWarrior Apr 29, 2020
7570d2a
Merge branch 'master' into data_loader
MekWarrior Apr 29, 2020
72da60c
fix more module import errors
MekWarrior Apr 29, 2020
9012fac
add first iteration of data loader
MekWarrior Apr 30, 2020
b94f653
data loader path assemble complete
MekWarrior Apr 30, 2020
47b1938
begin load function
MekWarrior May 1, 2020
5da6564
add mongodb with docker compose
MekWarrior May 5, 2020
dbb844d
fixed docker compose implementation
MekWarrior May 5, 2020
947dc63
basic read of metadata json working
MekWarrior May 5, 2020
d53d943
mongo db class
MekWarrior May 6, 2020
65b521c
add polish and stub out test file
MekWarrior May 6, 2020
079a233
first data loader complete
MekWarrior May 11, 2020
84efead
moved mongodb to new repo
MekWarrior May 11, 2020
cb9c93d
refine variable names and types
MekWarrior May 11, 2020
97a8d03
first version of universal data loader
MekWarrior May 12, 2020
f8b4688
Test related fixes and repo updates
MekWarrior May 13, 2020
9cad1b0
finish stubbing out tests for the data loader
MekWarrior May 13, 2020
b12577f
update testing
MekWarrior May 13, 2020
5871a4d
add testing for python 3.6
MekWarrior May 13, 2020
0de56fb
revise testing to more closely match deepcell style
MekWarrior May 13, 2020
6fb32d5
typo
MekWarrior May 13, 2020
63b08cf
remove testing for python 3.5
MekWarrior May 13, 2020
caa97f2
make test related adjustments
MekWarrior May 13, 2020
69306fe
clean up vocab check
MekWarrior May 14, 2020
8f9f51d
improve vocab check
MekWarrior May 14, 2020
b7a75c5
remove references to mongodb (to be added back in next PR)
MekWarrior May 14, 2020
3e69505
Clean up notebook
MekWarrior May 14, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 5 additions & 7 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
sudo: false
dist: trusty
dist: xenial

git:
depth: false
Expand All @@ -8,16 +7,15 @@ language: python

python:
- 3.6

cache: pip
- 3.7

install:
- pip install -r requirements.txt
- travis_retry pip install -r requirements.txt --progress-bar off
# install testing requirements
- pip install pytest==5.2.0 pytest-cov==2.5.1 pytest-pep8 coveralls
- travis_retry pip install -r requirements-test.txt --progress-bar off

script:
- python -m pytest --pep8 --cov=caliban_toolbox caliban_toolbox
- python -m pytest --cov=caliban_toolbox --pep8

after_success:
- coveralls
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.6
FROM python:3.7
MekWarrior marked this conversation as resolved.
Show resolved Hide resolved

# System maintenance
RUN apt-get update && apt-get install -y \
Expand Down
23 changes: 22 additions & 1 deletion caliban_toolbox/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Data Engineering Toolbox for DeepCell"""
"""Data Engineering Toolbox for Caliban"""

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

from caliban_toolbox import pre_annotation
# from caliban_toolbox import tracking
from caliban_toolbox import utils
from caliban_toolbox import aws_functions
from caliban_toolbox import figure_eight_functions
from caliban_toolbox import log_file
from caliban_toolbox import relabel
from caliban_toolbox import reshape_data

from caliban_toolbox.pre_annotation import *
# from caliban_toolbox.tracking import *
from caliban_toolbox.utils import *
MekWarrior marked this conversation as resolved.
Show resolved Hide resolved

del absolute_import
del division
del print_function
9 changes: 4 additions & 5 deletions caliban_toolbox/aws_functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,17 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
import sys
import boto3
import os
import sys
import threading
import re

from urllib.parse import urlencode
import boto3

import numpy as np
from urllib.parse import urlencode
from getpass import getpass

from caliban_toolbox.utils.utils import get_img_names, list_npzs_folder
from caliban_toolbox.utils.misc_utils import list_npzs_folder


# Taken from AWS Documentation
Expand Down
36 changes: 36 additions & 0 deletions caliban_toolbox/pre_annotation/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Copyright 2016-2020 The Van Valen Lab at the California Institute of
# Technology (Caltech), with support from the Paul Allen Family Foundation,
# Google, & National Institutes of Health (NIH) under Grant U24CA224309-01.
# All rights reserved.
#
# Licensed under a modified Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.github.com/vanvalenlab/caliban-toolbox/LICENSE
#
# The Work provided may be used for non-commercial academic purposes only.
# For any other use of the Work, including commercial use, please contact:
# vanvalenlab@gmail.com
#
# Neither the name of Caltech nor the names of its contributors may be used
# to endorse or promote products derived from this software without specific
# prior written permission.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Caliban Toolbox Pre-Annotation Module"""

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

from caliban_toolbox.pre_annotation import data_loader

del absolute_import
del division
del print_function