-
Notifications
You must be signed in to change notification settings - Fork 30
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Initial commit with README. Data available in davidstutz/caffe-tools-…
…data.
- Loading branch information
0 parents
commit 60aaa00
Showing
53 changed files
with
6,893 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,124 @@ | ||
| # Caffe-Tools | ||
|
|
||
| Tools and examples for pyCaffe, including: | ||
|
|
||
| * LMDB input and output and conversion from/to CSV and image files; | ||
| * monitoring the training process including error, loss and gradients; | ||
| * on-the-fly data augmentation; | ||
| * custom Python layers. | ||
|
|
||
| The data used for the examples can either be generated manually, see the documentation | ||
| or corresponding files in `examples`, or downloaded from [davidstutz/caffe-tools-data](https://github.com/davidstutz/caffe-tools-data). | ||
|
|
||
| Also see the corresponding blog articles at [davidstutz.de](http://davidstutz.de). | ||
|
|
||
| ## Examples | ||
|
|
||
| The provided examples include: | ||
|
|
||
| * [MNIST](http://yann.lecun.com/exdb/mnist/): [examples/mnist.py](examples/mnist.py) | ||
| * [Iris](https://archive.ics.uci.edu/ml/datasets/Iris): [examples/iris.py](examples/iris.py) | ||
| * [Cifar10](https://www.cs.toronto.edu/~kriz/cifar.html): [examples/cifar10.py](examples/cifar10.py.py) | ||
| * [BSDS500](https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html): [examples/bsds500.py](examples/bsds500.py) | ||
|
|
||
| Note that the BSDS500 example is **work in progress**! The corresponding data can | ||
| be downloaded from [davidstutz/caffe-tools-data](https://github.com/davidstutz/caffe-tools-data). | ||
| See the instructions in the corresponding files for details. | ||
|
|
||
| ## Resources | ||
|
|
||
| Some resources I found usefl while working with Caffe: | ||
|
|
||
| * Installation: | ||
| * http://stackoverflow.com/questions/31395729/how-to-enable-multithreading-with-caffe/31396229 | ||
| * https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN-3) | ||
| * https://github.com/BVLC/caffe/wiki/Ubuntu-16.04-or-15.10-Installation-Guide | ||
| * https://gist.github.com/titipata/f0ef48ad2f0ebc07bcb9 | ||
| * https://github.com/asampat3090/caffe-ubuntu-14.04 | ||
| * https://github.com/mrgloom/Caffe-snippets | ||
| * GitHub Repositories for pyCaffe: | ||
| * https://github.com/nitnelave/pycaffe_tutorial | ||
| * https://github.com/pulkitag/pycaffe-utils | ||
| * https://github.com/DeeperCS/pycaffe-mnist | ||
| * https://github.com/swift-n-brutal/pycaffe_utils | ||
| * https://github.com/jimgoo/caffe-oxford102 | ||
| * https://github.com/ruimashita/caffe-train | ||
| * https://github.com/roseperrone/video-object-detection | ||
| * https://github.com/pecarlat/caffeTools | ||
| * https://github.com/donnemartin/data-science-ipython-notebooks | ||
| * https://github.com/jay-mahadeokar/pynetbuilder | ||
| * https://github.com/adilmoujahid/deeplearning-cats-dogs-tutorial | ||
| * https://github.com/Franck-Dernoncourt/caffe_demos | ||
| https://github.com/koosyong/caffestudy | ||
| Issues: | ||
| * https://github.com/BVLC/caffe/issues/3651 (solverstate) | ||
| * https://github.com/BVLC/caffe/issues/1566 | ||
| * https://github.com/BVLC/caffe/pull/3082/files (snapshot) | ||
| * https://github.com/BVLC/caffe/issues/1257 (net surgery on solver net) | ||
| * https://github.com/BVLC/caffe/issues/409 (net diverges, loss = NaN) | ||
| * https://github.com/BVLC/caffe/issues/1168 (pyCaffe example incldued) | ||
| * https://github.com/BVLC/caffe/issues/462 (pyCaffe example incldued) | ||
| * https://github.com/BVLC/caffe/issues/2684 (change batch size) | ||
| * https://github.com/rbgirshick/py-faster-rcnn/issues/77 (load solverstate) | ||
| * https://github.com/BVLC/caffe/issues/2116 (Caffe LMDB float data) | ||
| * LMDB: | ||
| * https://lmdb.readthedocs.io/en/release/ | ||
| * http://research.beenfrog.com/code/2015/12/30/write-read-lmdb-example.html | ||
| * http://deepdish.io/2015/04/28/creating-lmdb-in-python/ | ||
| * https://github.com/BVLC/caffe/issues/3959 | ||
| * Tutorials/Blogs: | ||
| * http://christopher5106.github.io/deep/learning/2015/09/04/Deep-learning-tutorial-on-Caffe-Technology.html | ||
| * http://www.alanzucconi.com/2016/05/25/generating-deep-dreams/#part2 | ||
| * http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/ | ||
| * Caffe Versions: | ||
| * https://github.com/kevinlin311tw/caffe-augmentation (on-the-fly data augmentation) | ||
| * https://github.com/ShaharKatz/Caffe-Data-Augmentation (data augmentation) | ||
| * 3D: | ||
| * https://github.com/faustomilletari/3D-Caffe | ||
| * https://github.com/wps712/caffe4video | ||
| * StackOverflow: | ||
| * http://stackoverflow.com/questions/33905326/caffe-training-without-testing (training without testing) | ||
| * http://stackoverflow.com/questions/38348801/caffe-hangs-after-printing-data-label (stuck at data -> label) | ||
| * http://stackoverflow.com/questions/35529078/how-to-predict-in-pycaffe (predicting in pyCaffe) | ||
| * http://stackoverflow.com/questions/35529078/how-to-predict-in-pycaffe/35572495#35572495 (testing from LMDB with transformer) | ||
| * http://stackoverflow.com/questions/37642885/am-i-using-lmdb-incorrectly-it-says-environment-mapsize-limit-reached-after-0-i (LMDB mapsize) | ||
| * http://stackoverflow.com/questions/31820976/lmdb-increase-map-size (LMDB mapsize) | ||
| * http://stackoverflow.com/questions/34092606/how-to-get-the-dataset-size-of-a-caffe-net-in-python/34117558 (dataset size) | ||
| * http://stackoverflow.com/questions/32379878/cheat-sheet-for-caffe-pycaffe (pyCaffe cheat sheet) | ||
| * http://stackoverflow.com/questions/38511503/how-to-compute-test-validation-loss-in-pycaffe (copying weights to test net) | ||
| * http://stackoverflow.com/questions/29788075/setting-glog-minloglevel-1-to-prevent-output-in-shell-from-caffe (slience GLOG logging in | ||
| * http://stackoverflow.com/questions/36108120/shuffle-data-in-lmdb-file | ||
| * http://stackoverflow.com/questions/36459266/caffe-python-manual-sgd | ||
| * Layers: | ||
| * http://installing-caffe-the-right-way.wikidot.com/start | ||
| * https://github.com/NVIDIA/DIGITS/tree/master/examples/python-layer | ||
| * https://github.com/BVLC/caffe/blob/master/examples/pycaffe/layers/pyloss.py | ||
| * https://github.com/BVLC/caffe/blob/master/examples/pycaffe/layers/pascal_multilabel_datalayers.py | ||
| * http://stackoverflow.com/questions/34549743/caffe-how-to-get-the-phase-of-a-python-layer/34588801#34588801 | ||
| * http://stackoverflow.com/questions/34996075/caffe-data-layer-example-step-by-step | ||
| * https://github.com/BVLC/caffe/issues/4023 | ||
| * https://codegists.com/code/caffe-python-layer/ | ||
| * https://codedump.io/share/CiQmhfC63OD0/1/pycaffe-how-to-create-custom-weights-in-a-python-layer | ||
| * http://stackoverflow.com/questions/34498527/pycaffe-how-to-create-custom-weights-in-a-python-layer | ||
| * https://github.com/gcucurull/caffe-conf-matrix/blob/master/python_confmat.py | http://gcucurull.github.io/caffe/python/deep-learning/2016/06/29/caffe-confusion-matrix/ | ||
|
|
||
| ## Documentation | ||
|
|
||
| Installing and running Sphinx (also see [davidstutz/sphinx-example](https://github.com/davidstutz/sphinx-example) for details): | ||
|
|
||
| $ sudo apt-get install python-sphinx | ||
| $ sudo pip install sphinx | ||
| $ cd docs | ||
| $ make html | ||
|
|
||
| ## License | ||
|
|
||
| Copyright (c) 2016 David Stutz All rights reserved. | ||
|
|
||
| Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: | ||
|
|
||
| * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. | ||
| * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. | ||
| * Neither the name of David Stutz nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. | ||
|
|
||
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,225 @@ | ||
| # Makefile for Sphinx documentation | ||
| # | ||
|
|
||
| # You can set these variables from the command line. | ||
| SPHINXOPTS = | ||
| SPHINXBUILD = python -c "import sys,sphinx;sys.exit(sphinx.main(sys.argv))" | ||
| PAPER = | ||
| BUILDDIR = build | ||
|
|
||
| # Internal variables. | ||
| PAPEROPT_a4 = -D latex_paper_size=a4 | ||
| PAPEROPT_letter = -D latex_paper_size=letter | ||
| ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source | ||
| # the i18n builder cannot share the environment and doctrees with the others | ||
| I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source | ||
|
|
||
| .PHONY: help | ||
| help: | ||
| @echo "Please use \`make <target>' where <target> is one of" | ||
| @echo " html to make standalone HTML files" | ||
| @echo " dirhtml to make HTML files named index.html in directories" | ||
| @echo " singlehtml to make a single large HTML file" | ||
| @echo " pickle to make pickle files" | ||
| @echo " json to make JSON files" | ||
| @echo " htmlhelp to make HTML files and a HTML help project" | ||
| @echo " qthelp to make HTML files and a qthelp project" | ||
| @echo " applehelp to make an Apple Help Book" | ||
| @echo " devhelp to make HTML files and a Devhelp project" | ||
| @echo " epub to make an epub" | ||
| @echo " epub3 to make an epub3" | ||
| @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" | ||
| @echo " latexpdf to make LaTeX files and run them through pdflatex" | ||
| @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" | ||
| @echo " text to make text files" | ||
| @echo " man to make manual pages" | ||
| @echo " texinfo to make Texinfo files" | ||
| @echo " info to make Texinfo files and run them through makeinfo" | ||
| @echo " gettext to make PO message catalogs" | ||
| @echo " changes to make an overview of all changed/added/deprecated items" | ||
| @echo " xml to make Docutils-native XML files" | ||
| @echo " pseudoxml to make pseudoxml-XML files for display purposes" | ||
| @echo " linkcheck to check all external links for integrity" | ||
| @echo " doctest to run all doctests embedded in the documentation (if enabled)" | ||
| @echo " coverage to run coverage check of the documentation (if enabled)" | ||
| @echo " dummy to check syntax errors of document sources" | ||
|
|
||
| .PHONY: clean | ||
| clean: | ||
| rm -rf $(BUILDDIR)/* | ||
|
|
||
| .PHONY: html | ||
| html: | ||
| $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html | ||
| @echo | ||
| @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." | ||
|
|
||
| .PHONY: dirhtml | ||
| dirhtml: | ||
| $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml | ||
| @echo | ||
| @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." | ||
|
|
||
| .PHONY: singlehtml | ||
| singlehtml: | ||
| $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml | ||
| @echo | ||
| @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." | ||
|
|
||
| .PHONY: pickle | ||
| pickle: | ||
| $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle | ||
| @echo | ||
| @echo "Build finished; now you can process the pickle files." | ||
|
|
||
| .PHONY: json | ||
| json: | ||
| $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json | ||
| @echo | ||
| @echo "Build finished; now you can process the JSON files." | ||
|
|
||
| .PHONY: htmlhelp | ||
| htmlhelp: | ||
| $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp | ||
| @echo | ||
| @echo "Build finished; now you can run HTML Help Workshop with the" \ | ||
| ".hhp project file in $(BUILDDIR)/htmlhelp." | ||
|
|
||
| .PHONY: qthelp | ||
| qthelp: | ||
| $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp | ||
| @echo | ||
| @echo "Build finished; now you can run "qcollectiongenerator" with the" \ | ||
| ".qhcp project file in $(BUILDDIR)/qthelp, like this:" | ||
| @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/caffe-tools.qhcp" | ||
| @echo "To view the help file:" | ||
| @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/caffe-tools.qhc" | ||
|
|
||
| .PHONY: applehelp | ||
| applehelp: | ||
| $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp | ||
| @echo | ||
| @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." | ||
| @echo "N.B. You won't be able to view it unless you put it in" \ | ||
| "~/Library/Documentation/Help or install it in your application" \ | ||
| "bundle." | ||
|
|
||
| .PHONY: devhelp | ||
| devhelp: | ||
| $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp | ||
| @echo | ||
| @echo "Build finished." | ||
| @echo "To view the help file:" | ||
| @echo "# mkdir -p $$HOME/.local/share/devhelp/caffe-tools" | ||
| @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/caffe-tools" | ||
| @echo "# devhelp" | ||
|
|
||
| .PHONY: epub | ||
| epub: | ||
| $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub | ||
| @echo | ||
| @echo "Build finished. The epub file is in $(BUILDDIR)/epub." | ||
|
|
||
| .PHONY: epub3 | ||
| epub3: | ||
| $(SPHINXBUILD) -b epub3 $(ALLSPHINXOPTS) $(BUILDDIR)/epub3 | ||
| @echo | ||
| @echo "Build finished. The epub3 file is in $(BUILDDIR)/epub3." | ||
|
|
||
| .PHONY: latex | ||
| latex: | ||
| $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex | ||
| @echo | ||
| @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." | ||
| @echo "Run \`make' in that directory to run these through (pdf)latex" \ | ||
| "(use \`make latexpdf' here to do that automatically)." | ||
|
|
||
| .PHONY: latexpdf | ||
| latexpdf: | ||
| $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex | ||
| @echo "Running LaTeX files through pdflatex..." | ||
| $(MAKE) -C $(BUILDDIR)/latex all-pdf | ||
| @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." | ||
|
|
||
| .PHONY: latexpdfja | ||
| latexpdfja: | ||
| $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex | ||
| @echo "Running LaTeX files through platex and dvipdfmx..." | ||
| $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja | ||
| @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." | ||
|
|
||
| .PHONY: text | ||
| text: | ||
| $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text | ||
| @echo | ||
| @echo "Build finished. The text files are in $(BUILDDIR)/text." | ||
|
|
||
| .PHONY: man | ||
| man: | ||
| $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man | ||
| @echo | ||
| @echo "Build finished. The manual pages are in $(BUILDDIR)/man." | ||
|
|
||
| .PHONY: texinfo | ||
| texinfo: | ||
| $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo | ||
| @echo | ||
| @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." | ||
| @echo "Run \`make' in that directory to run these through makeinfo" \ | ||
| "(use \`make info' here to do that automatically)." | ||
|
|
||
| .PHONY: info | ||
| info: | ||
| $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo | ||
| @echo "Running Texinfo files through makeinfo..." | ||
| make -C $(BUILDDIR)/texinfo info | ||
| @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." | ||
|
|
||
| .PHONY: gettext | ||
| gettext: | ||
| $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale | ||
| @echo | ||
| @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." | ||
|
|
||
| .PHONY: changes | ||
| changes: | ||
| $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes | ||
| @echo | ||
| @echo "The overview file is in $(BUILDDIR)/changes." | ||
|
|
||
| .PHONY: linkcheck | ||
| linkcheck: | ||
| $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck | ||
| @echo | ||
| @echo "Link check complete; look for any errors in the above output " \ | ||
| "or in $(BUILDDIR)/linkcheck/output.txt." | ||
|
|
||
| .PHONY: doctest | ||
| doctest: | ||
| $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest | ||
| @echo "Testing of doctests in the sources finished, look at the " \ | ||
| "results in $(BUILDDIR)/doctest/output.txt." | ||
|
|
||
| .PHONY: coverage | ||
| coverage: | ||
| $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage | ||
| @echo "Testing of coverage in the sources finished, look at the " \ | ||
| "results in $(BUILDDIR)/coverage/python.txt." | ||
|
|
||
| .PHONY: xml | ||
| xml: | ||
| $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml | ||
| @echo | ||
| @echo "Build finished. The XML files are in $(BUILDDIR)/xml." | ||
|
|
||
| .PHONY: pseudoxml | ||
| pseudoxml: | ||
| $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml | ||
| @echo | ||
| @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." | ||
|
|
||
| .PHONY: dummy | ||
| dummy: | ||
| $(SPHINXBUILD) -b dummy $(ALLSPHINXOPTS) $(BUILDDIR)/dummy | ||
| @echo | ||
| @echo "Build finished. Dummy builder generates no files." |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| #!/bin/bash | ||
| sphinx-apidoc -f -o source/ ../tools/ | ||
| sphinx-apidoc -f -o source/ ../tests/ | ||
| sphinx-apidoc -f -o source/ ../examples/ | ||
| make html |
Oops, something went wrong.