Skip to content
Branch: master
Clone or download
drougge and berkeman [dataset_sort] Work with None/NaN values.
None already worked for some types, but broke if run on py3.
Now both work more or less properly. (For performance reasons they
sort as the smallest/largest comparable type instead of doing a
full comparison wrapper.)
Latest commit 3b8e58c May 17, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
conf [conf template] removed obsolete "hash_override" Mar 19, 2019
standard_methods [dataset_sort] Work with None/NaN values. May 17, 2019
test_methods [dataset_sort] Work with None/NaN values. May 17, 2019
urd [urd] logfileversion 3 May 15, 2018
.gitignore [gitignore] Remove all the old irrelevant stuff. Feb 15, 2018
LICENSE +LICENSE Apr 20, 2018
OVERVIEW.txt [OVERVIEW] Should have test_methods in it. Feb 26, 2019
README.md [README] + Home page Aug 21, 2018
autoflush.py +LICENSE Jun 22, 2017
automata_common.py [automata_common] print_profile reports relative time too Dec 30, 2018
automatarunner.py [automatarunner] Fix minimal tb on python2. Mar 3, 2018
blob.py +LICENSE Jun 22, 2017
chaining.py +LICENSE Jun 22, 2017
compat.py Handle when an analysis process dies badly. Mar 5, 2018
configfile.py s/workspace/workdir/ ONLY where visible to user Feb 13, 2018
control.py [control removed get_current_workspace() Dec 30, 2018
daemon.py [ctrl] fix jobid-LATEST Jan 25, 2019
database.py +LICENSE Jun 22, 2017
dataset.py [dataset] Don't require tuples in set_minmax. May 17, 2019
dependency.py +LICENSE Jun 22, 2017
deptree.py +LICENSE Jun 22, 2017
dispatch.py +LICENSE Jun 22, 2017
dscmdhelper.py s/workspace/workdir/ ONLY where visible to user Feb 13, 2018
dsgrep.py +LICENSE Jun 22, 2017
dsinfo.py String types can contain any character now. Mar 15, 2019
extras.py [extras] removed printresult() Mar 19, 2019
g.py +LICENSE Jun 22, 2017
gzwrite.py String types can contain any character now. Mar 15, 2019
jobid.py +LICENSE Jun 22, 2017
launch.py [status] Add an update_top_status for ananlysis. May 17, 2019
methods.py [methods] Better error for missing __init__.py Feb 22, 2018
report.py [report] Use compat.uni on everything. May 17, 2019
runner.py [runner] Don't die silently when _receiver fails. May 17, 2019
safe_pool.py +LICENSE Jun 22, 2017
setupfile.py +LICENSE Jun 22, 2017
sourcedata.py String types can contain any character now. Mar 15, 2019
status.py [status] Add an update_top_status for ananlysis. May 17, 2019
status_messaging.py [status] Add an update_top_status for ananlysis. May 17, 2019
subjobs.py +LICENSE Jun 22, 2017
unixhttp.py +LICENSE Jun 22, 2017
web.py +LICENSE Jun 22, 2017
workarounds.py [workarounds] stdin and stdout are usually the same. Mar 22, 2019
workspace.py [daemon] initiate all unknown workdirs in configfile Dec 30, 2018

README.md

This is the main Accelerator repository.

Use and Purpose

This is one of three repositories that are intended to be used together:

  1. https://github.com/eBay/accelerator-project_skeleton
  2. https://github.com/eBay/accelerator-gzutil
  3. https://github.com/eBay/accelerator

The purpose of the Accelerator project is to allow for fast data processing with big data. Extensive documentation on the purpose and how to use the Accelerator projects is covered in the following resources:

Reference Manual
Installation Manual
Home Page

Build and Runtime Environment

The Accelerator projects has been built, tested, and runs on:

  • Ubuntu16.04 and Debian 9,
  • FreeBSD 11.1

but is in no way limited to these systems or versions.

Installation

  1. Clone the https://github.com/eBay/accelerator-project_skeleton repository.

  2. Install dependencies. On Debian and Ubuntu

    sudo apt-get install build-essential python-dev python3-dev zlib1g-dev git virtualenv

  3. Run the setup script

    cd accelerator-project_skeleton
    ./init.py
    

    Please read and modify this script according to your needs.

  4. Done. The Accelerator is now ready for use.

The init.py script will clone both the accelerator-gzutil and the main accelerator repositories. The gzutil library will be set up in virtual environments for Python2 as well as Python3, and the Accelerator will be set up as a git submodule to the project_skeleton repository.

License

Copyright 2017-2018 eBay Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

You can’t perform that action at this time.