westurner/tools
Tools
See Also: https://westurner.github.io/wiki/projects#tools
Packages
A software package is an archive of files with a manifest that lists the files included. Often, the manifest contains file checksums and a signature.
Many packaging tools make a distinction between source and/or binary packages.
Some packaging tools provide configuration options for:
- Scripts to run when packaging
- Scripts to run at install time
- Scripts to run at uninstal time
- Patches to apply to the "vanilla" source tree, as might be obtained from a version control repository
There is a package maintainer whose responsibilities include:
- Testing new upstream releases
- Vetting changes from release to release
- Repackaging upstream releases
- Signing new package releases
Packaging lag refers to how long it takes a package maintainer to repackage upstream releases for the target platform(s).
Anaconda
Homepage: https://store.continuum.io/cshop/anaconda/
Docs: https://docs.continuum.io/anaconda/
Anaconda is a maintained distribution of Conda
packages for many languages; especially Python
.
Note
https://en.wikipedia.org/wiki/Anaconda_(installer) (1999) is the installer for RPM
-based Linux
distributions; which is also written in Python
(and C
).
APT
Homepage: https://alioth.debian.org/projects/apt
Docs: https://wiki.debian.org/Apt
Docs: https://www.debian.org/doc/manuals/debian-reference/ch02.en.html
Docs: https://www.debian.org/doc/manuals/apt-howto/
Docs: https://wiki.debian.org/SecureApt
Src: git git://anonscm.debian.org/git/apt/apt.git
IRC: irc://irc.debian.org/debian-apt
APT ("Advanced Packaging Tool") is the core of Debian
package management.
- An APT package repository serves
DEB
packages created withDpkg
. An APT package repository can be accessed from a local filesystem or over a network protocol ("apt transports") like HTTP, HTTPS, RSYNC, FTP, and BitTorrent (debtorrent).
An example of APT usage (e.g. to maintain an updated
Ubuntu
Linux
system):
apt-get update
apt-get upgrade
apt-get dist-upgrade
apt-cache show bash
apt-get install bash
apt-get --help
man apt-get
man sources.list
AUR
AUR (Arch
User Repository) contains PKGBUILD
packages which can be installed by pacman
.
Bower
Homepage: https://www.bower.io/
Src: https://github.com/bower/bower
Bower is "a package manager for the web" (Javascript
packages) built on NPM
.
BUILD
Docs: https://pantsbuild.github.io/build_dictionary.html
Docs: https://pantsbuild.github.io/options_reference.html
A BUILD
file describes a Pants Build
build.
Cabal
Docs: https://hackage.haskell.org/
Docs: https://www.haskell.org/cabal/users-guide/
Docs: https://www.haskell.org/cabal/release/cabal-latest/doc/API/Cabal/
Cabal is a package manager for Haskell
packages.
Hackage is the community Cabal package index: https://hackage.haskell.org/
Conda Package
Conda
Docs: https://conda.io/en/latest/
Src: git https://github.com/conda/conda
PyPI: https://pypi.python.org/pypi/conda
Conda is a package build, environment, and distribution system written in Python
to install packages written in any language.
- Conda was originally created for the Anaconda Python Distribution, which installs packages written in
Python
, R,Javascript
,Ruby
,C
,Fortran
- Conda packages are basically tar archives with build, and optional link/install and uninstall scripts.
conda-build
generates conda packages from conda recipes with ameta.yaml
, abuild.sh
, and/or abuild.bat
.- Conda recipes reference and build from a source package URI OR a
VCS <vcs>
URI and revision; and/or custombuild.sh
orbuild.bat
scripts. conda skeleton
can automatically create conda recipes fromPyPI
(Python),CRAN
(R), andCPAN
(Perl)conda skeleton
-generated recipes can be updated with additional metadata, scripts, and source URIs (as separate patches or consecutive branch commits of e.g. a conda-recipes repository in order to get a diff of the skeleton recipe and the current recipe).- Conda (and
Anaconda
) packages are hosted by https://binstar.org, which hosts free public and paid private Conda packages.- Anaconda Server is an internal "Private, Secure Package Repository" that "supports over 100 different repositories, including PyPI, CRAN, conda, and the Anaconda repository."
To create a fresh conda env:
# Python 2.7
conda create -n science --yes python readline conda-env
# Python 3.X
conda create -n science3 --yes python=3 readline conda-env
Work on a conda env:
source activate exmpl2
conda list
source deactivate
conda-env
writes to and creates environments from environment.yml
files which list conda and pip
packages.
Work with conda envs and environment.yml
files:
# Install conda-env globally (in the "root" conda environment)
conda install -n root conda-env
# Create a conda environment with ``conda-create`` and install conda-env
conda create -n science python=3 readline conda-env pip
# Install some things with conda (and envs/science/bin/pip)
# https://github.com/westurner/notebooks/blob/gh-pages/install.sh
conda search pandas; conda info pandas
conda install blaze dask bokeh odo \
sqlalchemy hdf5 h5py \
scikit-learn statsmodels \
beautiful-soup lxml html5lib pandas qgrid \
ipython-notebook
pip install -e git+https://github.com/rdflib/rdflib@master#egg=rdflib
pip install arrow sarge structlog
# Export an environment.yml
#source deactivate
conda env export -n science | tee environment.yml
# Create an environment from an environment.yml
conda env create -n projectname -f ./environment.yml
To install a conda package from a custom channel:
- https://www.pydanny.com/building-conda-packages-for-multiple-operating-systems.html
- https://github.com/conda/conda-recipes/tree/master/cookiecutter
- https://binstar.org/pydanny/cookiecutter
conda install -c pydanny cookiecutter # OR pip install cookiecutter
The conda-forge custom channel packages are built with continuous
integration
on multiple platforms:
Sources:
- https://github.com/conda
- https://github.com/conda/conda -- conda
- https://github.com/ContinuumIO/pycosat -- pycosat SAT solver
- https://github.com/conda/conda-env -- conda-env (the
conda env
command) - https://github.com/conda/conda-build -- conda-build (the
conda build
command) - https://github.com/conda/conda-recipes -- Community-maintained conda recipes (which users may build and
maintain <packages>
in https://binstar.org package repositories)
See also: Anaconda
, conda-forge
(conda-smithy
)
conda-forge
Src: https://github.com/conda-forge
Src: https://github.com/conda-forge/feedstocks
Src: https://github.com/conda-forge/feedstocks/tree/master/feedstocks
Src: https://github.com/conda-forge/staged-recipes
DockerHub: https://hub.docker.com/r/condaforge/linux-anvil
Docs: https://conda-forge.github.io/docs/
Docs: https://conda-forge.github.io/docs/recipe.html
- https://conda-forge.github.io/#add_recipe
- conda-smithy
- meta.yaml
- Docs: numpy x.x: https://conda-forge.github.io/docs/meta.html#building-against-numpy
- circle.yml
- .travis.yml
- appveyor.yml
- conda-forge.yml
- run_docker_build.sh https://github.com/conda-forge/staged-recipes/blob/master/scripts/run_docker_build.sh
- bootstrap-obvious-ci-and-miniconda.py https://github.com/conda-forge/conda-smithy/blob/master/bootstrap-obvious-ci-and-miniconda.py
# create a conda package recipe from a pypi package
cd $VIRTUAL_ENV/src
conda skeleton pypi jupyterthemes
ls -ld jupyterthemes/
edit jupyterthemes/meta.yaml
# - git repo tags || pypi releases
# create a conda-forge feedstock from a conda recipe
## https://github.com/conda-forge/conda-smithy#making-a-new-feedstock
cd $VIRTUAL_ENV/src
ls -ld jupyterthemes
conda-smithy init jupyterhemes
ls jupyterthemes-feedstock/
# build a conda-forge feedstock with docker
# FROM condaforge/linux-anvil
cat ./scripts/run_docker_build.sh
./scripts/run_docker_build.sh
./ci_support/run_docker_build.sh
conda-smithy
Docs: https://github.com/conda-forge/conda-smithy#making-a-new-feedstock
DEB
DEB is the Debian
software package format.
DEB packages are built with dpkg
and often hosted in an APT
package repository.
Dpkg
Homepage: https://wiki.debian.org/Teams/Dpkg
Docs: https://en.wikipedia.org/wiki/Debian_build_toolchain
Docs: https://www.debian.org/doc/manuals/debian-faq/ch-pkg_basics.en.html
Docs: https://www.debian.org/doc/manuals/debian-faq/ch-pkgtools.en.html
Docs:
Dpkg is a collection of tools for creating and working with DEB
packages.
dnf
Homepage: https://fedoraproject.org/wiki/DNF
Homepage: https://fedoraproject.org/wiki/Features/DNF
Src: git https://github.com/rpm-software-management/dnf
Docs: https://dnf.readthedocs.io/en/latest/
Docs: https://github.com/rpm-software-management/dnf/wiki
Docs: https://rpm-software-management.github.io/dnf-plugins-core/
dnf is a an open source package manager written in Python
.
- dnf was introduced in
Fedora
18. - dnf is the default package manager in
Fedora
22; replacingyum
.- [ ]
yum
errors if TODO package is installed (*salt
provider) - [ ]
repoquery
redirects with an error todnf repoquery
- See
dnf help
(andman dnf
)
- [ ]
- dnf integrates with the Anaconda system installer.
dnf
supports Delta RPM packages (DRPM), which often significantly reduce the required amount of network transfer required to regularly retrieve and upgrade to the latest repository packages.
ebuild
Homepage: https://wiki.gentoo.org/wiki/Ebuild
Docs: https://devmanual.gentoo.org/quickstart/
Docs: https://devmanual.gentoo.org/ebuild-writing/
Docs: https://devmanual.gentoo.org/ebuild-writing/file-format/
Docs: https://devmanual.gentoo.org/ebuild-writing/variables/
Docs: https://devmanual.gentoo.org/ebuild-writing/use-conditional-code/
Docs: https://wiki.gentoo.org/wiki/Submitting_ebuilds
ebuild is a software package definition format.
- ebuilds are like special
bash
scripts. - ebuilds have
USE
flags for specifying build features. Gentoo
is built from ebuild package definitions stored in GentooPortage
.Portage
packages are built from ebuilds.- The
emerge
Portage
command installs ebuilds.
emerge
Homepage: https://wiki.gentoo.org/wiki/Portage#emerge
Src: git https://github.com/gentoo/portage
Docs: https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification
Docs: https://projects.gentoo.org/pms/6/pms.html
emerge
is the primary CLI tool used for installing packages built from ebuilds <ebuild>
[from Portage
].
fpm
Src: https://github.com/jordansissel/fpm
Docs: https://github.com/jordansissel/fpm/wiki/
Docs: https://github.com/jordansissel/fpm/wiki/PackageMakeInstall
fpm (effing package management) is a tool for building many types of software packages from many other types of software packages (e.g. DEB
. RPM
, Python Packages
); often more easily than working with the actual package manager.
- fpm package source types include: dir rpm gem python empty tar deb cpan npm osxpkg pear pkgin virtualenv zip.
- fpm target package types include: rpm deb solaris puppet dir osxpkg p5p puppet sh tar zip
Brew
Homebrew
Homepage: https://brew.sh/
Src: https://github.com/Homebrew/brew
Homebrew is a package manager (brew
) for OSX
.
NPM
Node Package Manager
Homepage: https://www.npmjs.org/
Src: https://github.com/npm/npm
Docs: https://docs.npmjs.com/files/package.json#files
NPM is a Javascript
package manager created for Node.js
.
- an NPM package is defined by a
package.json
JSON
file. - NPM packages are installed with the
npm
CLI utility. Bower
builds upon NPM.
NuGet
NuGet is an open source package manager for Windows
.
- Chocolatey maintains variously updated packages for various windows programs: https://chocolatey.org/
- An example list of Chocolatey NuGet packages as a
PowerShell
script: https://gist.github.com/westurner/10950476
- An example list of Chocolatey NuGet packages as a
pacman
Homepage: https://www.archlinux.org/pacman/
Docs: https://wiki.archlinux.org/index.php/Pacman
Pacman is an open source package manager which installs .pkg.tar.xz
files for Arch
Linux.
PEX
Src: https://github.com/pantsbuild/pex
PyPI: https://pypi.python.org/pypi/pex
Docs: https://pex.readthedocs.io/en/stable/
PEX (Python Executable) is a ZIP
-based software package archive format with an executable header.
Pants
creates PEX packages.
PKGBUILD
Docs: https://www.archlinux.org/pacman/PKGBUILD.5.html
Docs: https://wiki.archlinux.org/index.php/Makepkg
Docs: https://wiki.archlinux.org/index.php/Creating_packages
PKGBUILD is a shell script containing the build information for an AUR
Arch
linux
software package.
Portage
Homepage: https://wiki.gentoo.org/wiki/Project:Portage
Docs: https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification
Docs: https://projects.gentoo.org/pms/6/pms.html
Portage is a package management and repository system written in Python
initially just for Gentoo
Linux
.
Emerge
installsebuilds <ebuild>
fromportage
.
Ports
A Ports collection contains Sources (e.g. archived releases and patch sets) and Makefiles <make>
designed to compile software packages
for particular operating systems <operating system>
distributions' kernel and standard libraries usually for a particular platform.
RPM
RPM (RPM Package Manager, RedHat
Package Manager) is a package
format and a set of commandline utilities written in C
and Perl
.
- RPM packages can be installed with
rpm
,yum
,dnf
. - RPM pacage can be built with tools like
rpmbuild
andfpm
- Python packages can be built into RPM packages with
setuptools' <setuptools>
bdist_rpm
,fpm
List contents of RPM packages (archives) with e.g.
less
andlesspipe
:less ~/path/to/local.rpm # requires lesspipe to be configured
- RPM Packages are served by and retrieved from repositories by tools like
yum
anddnf
:- Local: directories of
RPM
packages and metadata - Network:
HTTP <http->
,HTTPS <https-->
,RSYNC
, FTP dnf
supports Delta RPM packages (DRPM), which often significantly reduce the required amount of network transfer required to regularly retrieve and upgrade to the latest repository packages.
- Local: directories of
Note
There's not yet a debtorrent
for RPM
, YUM
, DNF
.
Egg
Python Egg
Python Packages
Download: https://pypi.python.org/simple/
Docs: https://packaging.python.org/en/latest/
Docs: https://packaging.python.org/en/latest/current/
Docs: https://packaging.python.org/en/latest/distributing/
Docs: https://packaging.python.org/en/latest/peps.html
Docs: https://packaging.python.org/en/latest/projects.html
Docs: https://packaging.python.org/en/latest/specifications/
Docs: https://pypaio.readthedocs.io/en/latest/roadmap/
A Python
Package is a collection of source code and package data files.
- Python packages have dependencies: they depend on other packages
- Python packages can be served from a package index
PyPI
is the community Python Package Index- A Python package is an archive of files (
.zip
(.egg
,.whl
),.tar
,.tar.gz
,) containing asetup.py
file containing a version string and metadata that is meant for distribution. - An source dist (
sdist
) package contains source code (every file listed in or matching a pattern in aMANIFEST.in
text file). - A binary dist (
bdist
,bdist_egg
,bdist_wheel
) is derived from an sdist and may be compiled and named for a specific platform. - sdists and bdists are defined by a
setup.py
file which contains a call to adistutils.setup()
orsetuptools.setup()
function. - The arguments to the
setup.py
function are things likeversion
,author
,author_email
, andhomepage
; in addition to package dependency strings required for the package to work (install_requires
), for tests to run (tests_require
), and for optional things to work (extras_require
). - A package dependency string can specify an exact version (
==
) or a greater-than (>=
) or less-than (<=
) requirement for each package. - Package names are looked up from an index server (
--index
), such asPyPI
, and or an HTML page (--find-links
) containing URLs containing package names, version strings, and platform strings. easy_install
(setuptools
) andpip
can install packages from: the local filesystem, a remote index server, or a local index server.easy_install
andpip
read theinstall_requires
(andextras_require
) attributes ofsetup.py
files contained in packages in order to resolve a dependency graph (which can contain cycles) and install necessary packages.
.
- PyPA Tool Recommendations
- PyPA Python Package PEPs
- PyPA Projects List
Note
JSON-LD-
for package metadata and environment build metadata could be helpful.
distutils
Distutils is a collection of tools for common packaging needs.
- Distutils is included in the
Python
standard library.
setuptools
Docs: https://pythonhosted.org/setuptools/
Src: https://github.com/pypa/setuptools
PyPI: https://pypi.python.org/pypi/setuptools
Setuptools is a Python package <python packages>
for working with other Python Packages
.
- Setuptools builds upon
distutils
- Setuptools is widely implemented
- Most Python packages are installed by setuptools (by
Pip
) - Setuptools can be installed by downloading
ez_setup.py
and then runningpython ez_setup.py
; or, setuptools can be installed with a system package manager (apt, yum) - Setuptools installs a script called
easy_install
which can be used to install packages from the local filesystem, a remote index server, a local index server, or an HTML page easy_install pip
installsPip
from PyPI- Like
easy_install
,Pip
installs python packages, with a number of additional configuration options Setuptools can build
RPM
andDEB
packages from python packages, with some extra configuration:python setup.py bdist_rpm --help python setup.py --command-packages=stdeb.command bdist_deb --help
Pip
Homepage: https://pip.pypa.io/
Docs: https://pip.pypa.io/en/latest/user_guide.html
Docs: https://pip.readthedocs.io/en/latest/
Src: git https://github.com/pypa/pip
Pypi: https://pypi.python.org/pypi/pip
IRC: #pypa
IRC: #pypa-dev
Pip is a tool for installing, upgrading, and uninstalling Python
packages.
pip help
pip help install
pip --version
sudo apt-get install python-pip
pip install --upgrade pip
pip install libcloud
pip install -r requirements.txt
pip uninstall libcloud
- Pip stands upon
distutils
andsetuptools
. - Pip retrieves, installs, upgrades, and uninstalls packages.
- Pip can list installed packages with
pip freeze
(andpip list
). - Pip can install packages as 'editable' packages (
pip install -e
) from version control repository URLs which must begin withvcs+
, end with#egg=<usuallythepackagename>
, and may contain an@vcstag
tag (such as a branch name or a version tag). - Pip installs packages as editable by first cloning (or checking out) the code to
./src
(or${VIRTUAL_ENV}/src
if working in avirtualenv
) and then runningsetup.py develop
. - Pip configuration is in
${HOME}/.pip/pip.conf
. - Pip can maintain a local cache of downloaded packages, which can lessen the load on package servers during testing.
- Pip skips reinstallation if a package requirement is already satisfied.
- Pip requires the
--upgrade
and/or--force-reinstall
options to be added to thepip install
command in order to upgrade or reinstall. - At the time of this writing, the latest stable pip version is
1.5.6
.
Warning
With Python
2, pip is preferable to setuptools
's easy_install
because pip installs backports.ssl_match_hostname
in order to validate HTTPS
certificates (by making sure that the certificate hostname matches the hostname from which the DNS resolved to).
Cloning packages from source repositories over ssh://
or https://
, either manually or with pip install -e
avoids this concern.
There is also a tool called peep
which requires considered-good SHA256 checksums to be specified for every dependency listed in a requirements.txt
file.
For more information, see: https://legacy.python.org/dev/peps/pep-0476/#python-versions
- Pip Requirements File
Plaintext list of packages and package URIs to install.
Requirements files may contain version specifiers (
pip >= 1.5
)Pip installs Pip Requirement Files:
pip install -r requirements.txt pip install --upgrade -r requirements.txt pip install --upgrade --user --force-reinstall -r requirements.txt
An example
requirements.txt
file:# install pip from the default index (PyPI) pip --index=https://pypi.python.org/simple --upgrade pip # Install pip 1.5 or greater from PyPI pip >= 1.5 # Git clone and install pip as an editable develop egg -e git+https://github.com/pypa/pip@1.5.X#egg=pip # Install a source distribution release from PyPI # and check the MD5 checksum in the URL https://pypi.python.org/packages/source/p/pip/pip-1.5.5.tar.gz#md5=7520581ba0687dec1ce85bd15496537b # Install a source distribution release from Warehouse https://warehouse.python.org/packages/source/p/pip/pip-1.5.5.tar.gz # Install an additional requirements.txt file -r requirements/more-requirements.txt
Peep
Peep works just like pip
, but requires SHA256
checksum hashes to be specified for each package in requirements.txt
file.
Python Package Index
PyPI
Docs: https://wiki.python.org/moin/CheeseShop
Docs: https://wiki.python.org/moin/CheeseShopDev
Homepage: https://pypi.python.org/pypi
Src: https://github.com/pypa/pypi-legacy
PyPI-legacy is the original Python Package Index. PyPI is now powered by Warehouse
Warehouse
Src: https://github.com/pypa/warehouse
Docs: https://warehouse.readthedocs.io/en/latest/
Warehouse is the "Next Generation Python Package Repository".
All packages uploaded to PyPI
are also available from Warehouse.
Devpi
Src: https://github.com/devpi/devpi
Issues: https://github.com/devpi/devpiissues
PyPI: https://pypi.python.org/pypi/devpi-server
PyPI: https://pypi.python.org/pypi/devpi-web
PyPI: https://pypi.python.org/pypi/devpi-client
Docs: https://doc.devpi.net/latest/
Devpi is a server and client solution for Python package <python
packages>
mirroring, hosting, and testing.
Python Wheel
Wheel
Docs: https://wheel.readthedocs.io/en/latest/
Src: https://github.com/pypa/wheel
PyPI: https://pypi.python.org/pypi/wheel
- Wheel is a newer, PEP-based standard (
.whl
) with a different metadata format, the ability to specify (JSON) digital signatures for a package within the package, and a number of additional speed and platform-consistency advantages. - Wheels can be uploaded to PyPI.
- Wheels are generally faster than traditional Python packages.
Packages available as wheels are listed at https://pythonwheels.com/.
Ruby Gem
RubyGems
Homepage: https://rubygems.org/
Docs: https://guides.rubygems.org/
Src: https://github.com/rubygems/rubygems
RubyGems is a package manager for Ruby
packages ("Gems").
Yum
Homepage: http://yum.baseurl.org/
Yum is a tool for installing, upgrading, and uninstalling RPM
packages.
Version Control Systems
Distributed Version Control Systems
Wikipedia: https://en.wikipedia.org/wiki/Distributed_revision_control
Version Control Systems (VCS) --- or Revision Control Systems (RCS) ---are designed to solve various problems in change management.
- VCS store code in a repository.
- Changes to one or more files are called changesets, commits, or revisions
- Changesets are comitted or checked into to a repository.
- Changesets are checked out from a repository
- Many/most VCS differentiate between the repository and a working directory, which is currently checked out to a specific changeset identified by a revision identifier; possibly with uncommitted local changes.
- A branch is forked from a line of development and then merged back in.
- Most projects designate a main line of development referred to as a trunk, master, or default branch.
- Many projects work with feature and release branches, which, ideally, eventually converge by being merged back into trunk. (see:
HubFlow
for an excellent example of branching) - Traditional VCS are centralized on a single point-of-failure.
- Some VCS have a concept of locking to prevent multiple peoples' changes from colliding
- Distributed Version Control Systems (DVCS) (can) clone all revisions of every branch of a repository every time. *
- DVCS changesets are pushed to a different repository
- DVCS changesets are pulled from another repository into a local clone or copy of a repository
- Teams working with DVCS often designate a central repository hosted by a project forge service like SourceForge, GNU Savannah, GitHub, or BitBucket.
- Contributors send patches which build upon a specific revision, which can be applied by a maintainer with commit access permissions.
- Contributors fork a new branch from a specific revision, commit changes, and then send a pull request, which can be applied by a maintainer with commit access permissions.
CVS
Homepage: https://savannah.nongnu.org/projects/cvs
Wikipedia: https://en.wikipedia.org/wiki/Concurrent_Versions_System
Docs: https://www.nongnu.org/cvs/#documentation
CVS (cvs
) is a centralized version control system (VCS) written in C
.
CVS predates most/many other VCS.
Subversion
Wikipedia: https://en.wikipedia.org/wiki/Apache_Subversion
Docs: https://subversion.apache.org/docs/
Docs: https://subversion.apache.org/quick-start
Src: svn https://svn.apache.org/repos/asf/subversion/trunk
Src: https://github.com/apache/subversion
Apache Subversion (svn
) is a centralized revision control system (VCS) written in C
.
To checkout a revision of a repository with svn
:
svn co https://svn.apache.org/repos/asf/subversion/trunk subversion
Bazaar
Homepage: https://bazaar.canonical.com/en/
Homepage: https://launchpad.net/bzr
Docs: http://doc.bazaar.canonical.com/en/
Docs: http://doc.bazaar.canonical.com/latest/en/mini-tutorial/index.html
Src: bzr lp:bzr
GNU Bazaar (bzr
) is a distributed revision control system (DVCS, RCS, VCS) written in Python
and C
.
https://launchpad.net hosts Bazaar repositories; with special support from the bzr
tool in the form of lp:
URIs like lp:bzr
.
To clone a repository with bzr
:
bzr branch lp:bzr
Git
Src: https://github.com/git/git
Homepage: https://git-scm.com/
Docs: https://git-scm.com/documentation
Docs: https://git-scm.com/book/en/
Docs: https://github.com/skwp/git-workflows-book
Docs: https://learnxinyminutes.com/docs/git/
Docs: https://learngitbranching.js.org/
Docs: https://www.leshenko.net/p/ugit/
Docs: https://github.com/westurner/dotfiles/blob/develop/etc/.gitconfig
Git (git
) is an open source distributed version control system for tracking a branching and merging repository of file revisions written in C
(DVCS, VCS, RCS).
To clone a repository with git
:
git clone https://github.com/git/git
cd ./git
git status; git remote -av; git reflog;
git help help; git help reflog
GitFlow
Docs: https://nvie.com/posts/a-successful-git-branching-model/
Docs: https://github.com/nvie/gitflow/wiki
Docs: https://github.com/nvie/gitflow/wiki/Command-Line-Arguments
Docs: https://github.com/nvie/gitflow/wiki/Config-values
GitFlow is a named branch workflow for git
with master
, develop
, feature
, release
, hotfix
, and support
branches (git flow
).
Gitflow branch names and prefixes are configured in .git/config
; the defaults are:
Branch Name | Description (and Code Labels) |
master |
Stable trunk (latest release) |
develop |
Development main line |
feature/<name> |
New features for the next release (e.g. ENH , PRF ) |
release/<name> |
In-progress release branches (e.g. RLS ) |
hotfix/<name> |
Fixes to merge to both master and develop (e.g. BUG , TST , DOC ) |
support/<name> |
"What is the 'support' branch?" |
Creating a new release with Git
and GitFlow
:
git clone ssh://git@github.com/westurner/dotfiles
# git checkout master
# git checkout -h
# git help checkout (man git-checkout)
# git flow [<cmd> -h]
# git-flow [<cmd> -h]
git flow init
## Update versiontag in .git/config to prefix release tags with 'v'
git config --replace-all gitflow.prefix.versiontag v
cat ./.git/config
# [gitflow "prefix"]
# feature = feature/
# release = release/
# hotfix = hotfix/
# support = support/
# versiontag = v
#
## feature/ENH_print_hello_world
git flow feature start ENH_print_hello_world
#git commit, commit, commit
git flow feature
git flow feature finish ENH_print_hello_world # ENH<TAB>
## release/0.1.0
git flow release start 0.1.0
#git commit (e.g. update __version__, setup.py, release notes)
git flow release finish 0.1.0
git flow release finish 0.1.0
git tag | grep 'v0.1.0'
HubFlow
Docs: https://datasift.github.io/gitflow/
Docs: https://datasift.github.io/gitflow/IntroducingGitFlow.html
Docs: https://datasift.github.io/gitflow/TheHubFlowTools.html
Docs: https://datasift.github.io/gitflow/GitFlowForGitHub.html
GitFlow is a named branch workflow for git
with master
, develop
, feature
, release
, hotfix
, and support
branches (git flow
).
HubFlow is a fork of GitFlow
that adds useful commands for working with Git
and GitHub pull requests.
HubFlow branch names and prefixes are configured in .git/config
; the defaults are:
Branch Name | Description (and Code Labels) |
master |
Stable trunk (latest release) |
develop |
Development main line |
feature/<name> |
New features for the next release (e.g. ENH , PRF ) |
release/<name> |
In-progress release branches (e.g. RLS ) |
hotfix/<name> |
Fixes to merge to both master and develop (e.g. BUG , TST , DOC ) |
Creating a new release with Git
and HubFlow
:
git clone ssh://git@github.com/westurner/dotfiles
# git checkout master
# git checkout -h
# git help checkout (man git-checkout)
# git hf [<cmd> -h]
# git-hf [<cmd> -h]
git hf init
## Update versiontag in .git/config to prefix release tags with 'v'
git config --replace-all hubflow.prefix.versiontag v
#cat .git/config # ...
# [hubflow "prefix"]
# feature = feature/
# release = release/
# hotfix = hotfix/
# support = support/
# versiontag = v
#
git hf update
git hf pull
git hf pull -h
## feature/ENH_print_hello_world
git hf feature start ENH_print_hello_world
#git commit, commit
git hf pull
git hf push
#git commit, commit
git hf feature finish ENH_print_hello_world # ENH<TAB>
## release/0.1.0
git hf release start 0.1.0
## commit (e.g. update __version__, setup.py, release notes)
git hf release finish 0.1.0
git hf release finish 0.1.0
git tag | grep 'v0.1.0'
The GitFlow HubFlow illustrations are very helpful for visualizing and understanding any DVCS workflow: https://datasift.github.io/gitflow/IntroducingGitFlow.html.
Hg
Mercurial
Homepage: https://www.mercurial-scm.org/
Docs: https://www.mercurial-scm.org/guide
Docs: https://book.mercurial-scm.org/
Src: hg https://www.mercurial-scm.org/repo/hg
Src: hg http://hg.intevation.org/mercurial
Mercurial (hg
) is a distributed revision control system written in Python
and C
(DVCS, VCS, RCS).
To clone a repository with hg
:
hg clone https://www.mercurial-scm.org/repo/hg
GitHub
Web: https://github.com/
Src: https://github.com/github
Docs: https://docs.github.com/en
Docs: https://docs.github.com/en/get-started/quickstart/git-and-github-learning-resources
Docs: https://docs.github.com/en/get-started/quickstart/github-glossary
Docs: https://skills.github.com/
Docs: https://github.com/skills
Docs: https://github.com/skills/introduction-to-github
Docs: https://github.com/skills/communicate-using-markdown
StatusPage: https://www.githubstatus.com/
Git
repos, Issues, Pull Requests, Wikis, Pages, Actions, Project Boards, Webhooks
- https://github.com/github/choosealicense.com
- https://docs.github.com/en/get-started/quickstart/hello-world
- https://docs.github.com/en/get-started/quickstart/github-flow
hubflow
is a fork ofgitflow
for use with GitHub Pull requests- https://github.com/topics/awesome
- https://github.com/devspace/awesome-github-templates
- https://github.com/stevemao/github-issue-templates/blob/master/must-open-issue-before-pr/PULL_REQUEST_TEMPLATE.md
- https://github.com/stevemao/github-issue-templates/blob/master/checklist2/PULL_REQUEST_TEMPLATE.md
- https://github.com/abhisheknaiidu/awesome-github-profile-readme
GitHub CI
GitHub Actions
Src: https://github.com/actions
Src: https://github.com/actions/runner
Src: https://github.com/actions/runner-images
Src: https://github.com/actions/deploy-pages
Src: https://github.com/actions/upload-pages-artifact
Src: https://github.com/actions/upload-artifact
Src: https://github.com/actions/upload-release-asset
Src: https://github.com/softprops/action-gh-release
Docs: https://docs.github.com/en/actions
Docs: https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions
Docs: https://docs.github.com/en/actions/quickstart#creating-your-first-workflow
Docs: https://github.com/actions/starter-workflows
- Src: https://github.com/nektos/act
- Run GitHub Actions locally; without GitHub Runner
GitHub Classroom
Web: https://education.github.com/
Docs: https://docs.github.com/en/education/manage-coursework-with-github-classroom/get-started-with-github-classroom/about-github-classroom#github-classroom-features
otter-grader
andnbgrader
autogradeJupyter
notebooks (in containers for safety) and can post grades to an LMS / LRS
GitLab
Web: https://gitlab.com/
Src: https://gitlab.com/gitlab-org
Src: https://gitlab.com/gitlab-org/gitlab
Issues: https://gitlab.com/groups/gitlab-org/-/issues
Docs: https://docs.gitlab.com/
Docs: https://docs.gitlab.com/ee/user/markdown.html#where-you-can-use-gitlab-flavored-markdown
Docs: https://docs.gitlab.com/ee/install/install_methods.html
Docs: https://about.gitlab.com/direction/maturity/#package
Docs: https://about.gitlab.com/handbook/
Docs: https://about.gitlab.com/handbook/markdown-guide/
Docs: https://about.gitlab.com/company/kpis/
Docs: https://about.gitlab.com/company/okrs/
StatusPage: https://status.gitlab.com/
- GitLab is written in
Ruby
,Go
, andJS
.
GitLab CI
GitLab CI is an open source Continuous Integration
system which runs commands in containers per tasks defined in a build YAML
file on project events like git push, new Pull Request branch, new Issue.
- gitlab-ci.yml
- GitLab CI precedes
GitHub
. Travis-CI precedes GitLab CI andGitHub Actions
. Bitten by Edgewall (Trac) precedes Travis CI.
Gitea
Web: https://gitea.io/en-us/
Src: https://github.com/go-gitea/gitea
Docs: https://docs.gitea.io/en-us/
Docs: https://try.gitea.io/api/swagger
Gitea is an open source project forge site written in Go
; with Git
repositories (repos), Wiki repos, Issues, Pull Requests; and Continuous Integration
to run the build script (YAML
) on events like git push, new_issue, and new_pr;
Package
repositories for released build artifacts, Container
repository
- https://docs.gitea.io/en-us/usage/automatically-linked-references/
- https://docs.gitea.io/en-us/installation/install-with-docker-rootless/
- https://docs.gitea.io/en-us/administration/https-setup/#using-acme-default-lets-encrypt
- https://docs.gitea.io/en-us/installation/upgrade-from-gitea/
- https://docs.gitea.io/en-us/administration/command-line/
- https://docs.gitea.io/en-us/usage/repo-mirror/
- https://github.com/maxkratz/github2gitea-mirror
- https://docs.gitea.io/en-us/usage/packages/overview/#supported-package-managers
- https://docs.gitea.io/en-us/usage/packages/container/
- https://docs.gitea.io/en-us/usage/packages/conda/
- https://docs.gitea.io/en-us/usage/packages/pypi/
- https://docs.gitea.io/en-us/usage/packages/npm/
- https://docs.gitea.io/en-us/administration/external-renderers/#example-jupyter-notebook
- Gitea supports file-attachments in issues, pull requests, and releases (tagged git commits)
- Forgejo is a fork of Gitea, like Gitea is a fork of Gogs (which is a clone of GitHub)
Gitea Actions
- Src: https://github.com/go-gitea/gitea/tree/main/modules/actions
- Src: https://github.com/go-gitea/gitea/tree/main/services/actions
- Src: https://gitea.com/gitea/act_runner
- Docs: https://docs.gitea.io/en-us/usage/usage/actions/overview/
- Docs: https://docs.gitea.io/en-us/usage/usage/actions/quickstart/#set-up-runner
- Docs: https://docs.gitea.io/en-us/usage/usage/actions/comparison/ w/
GitHub Actions
- Docs: https://blog.gitea.io/2023/03/hacking-on-gitea-actions/
Project Templates
cookiecutter
(Python, [...])yeoman
(JS, [...])jinja2
(salt
,ansible
configuration management
Exemplar Projects
https://www.python.org/dev/peps/
"PEP 0 -- Index of Python Enhancement Proposals (PEPs)"https://www.python.org/dev/peps/pep-0012/
"PEP 0012 -- SamplereStructuredText
PEP Template" (Python
)
https://github.com/ipython/ipython/wiki/IPEPs:-IPython-Enhancement-Proposals
"IPEPs: IPython Enhancement Proposals"https://github.com/ipython/ipython/wiki/IPEP-0%3A-IPEP-Template
"IPEP 0: IPEP Template" (IPython
,Jupyter
)- https://github.com/jupyter/roadmap
github.com/westurner/wiki: a GitHub Sphinx Wiki
Home: https://westurner.github.io/wiki/
Wiki: https://github.com/westurner/wiki/wiki
Src: https://github.com/westurner/wiki
Src: https://github.com/westurner/wiki.wiki.git
Issues: https://github.com/westurner/wiki/issuesmake help # - build the docs w/ sphinx # - push to both branches # - gh-pages from eg _build/html, _build/singlehtml make docs push gh-pages
github.com/rdfjs/rdfjs.org: a GitHub Project
seeAlso:
- https://wrdrd.github.io/docs/consulting/software-development#agile
- https://wrdrd.github.io/docs/consulting/team-building#the-same-page
cookiecutter
Src: git https://github.com/audreyr/cookiecutter
PyPI: https://pypi.python.org/pypi/cookiecutter
Docs: https://cookiecutter.readthedocs.io/en/latest/
Docs: https://cookiecutter.readthedocs.io/en/latest/usage.html
Docs: https://cookiecutter.readthedocs.io/en/latest/tutorials.html#create-your-very-own-cookiecutter
Cookiecutter creates projects (files and directories) from project templates written in Jinja2
for projects written in Python
and other languages.
https://cookiecutter.readthedocs.io/en/latest/readme.html#available-cookiecutters
List of Cookiecutter generators
https://github.com/audreyr/cookiecutter-pypackage
Cookiecutter template for a
Python Package <Python packages>
(e.g.setup.py
,docs/
,README.rst
)https://github.com/pydanny/cookiecutter-django
Cookiecutter template for a Django project w/ Bootstrap, AngularJS,
Docker
,https://github.com/pydanny/cookiecutter-djangopackage
Cookiecutter template for reusable Django packages (installable apps).
https://github.com/openstack-dev/cookiecutter
Cookiecutter template for
OpenStack
Python Package <Python packages>
projects (*pip
*, pbr, tox,sphinx
)https://github.com/openstack-dev/specs-cookiecutter
Cookiecutter template for
OpenStack
specs projects (pbr, tox,sphinx
)
yeoman
Homepage: https://yeoman.io/
Src: git https://github.com/yeoman/yeoman
Src: https://github.com/yeoman
NPM: https://www.npmjs.com/package/yo
NPMPkg:
yo
Docs: https://yeoman.io/learning/
Docs: https://yeoman.io/codelab/
Docs: https://yeoman.io/authoring
Docs: https://yeoman.io/learning/resources.html
https://github.com/yeoman/generator-generator
Generate a Yeoman generator (
./authoring
).-
List of Yeoman generators
https://github.com/yeoman/generator-angular
AngularJS 1 Yeoman generator (
bower
, karma tests,CoffeeScript
,TypeScript
)https://github.com/diegonetto/generator-ionic
Apache Cordova mobile app w/ Ionic (AngularJS 1,
grunt
)https://stackoverflow.com/questions/29649578/available-yeoman-generator-for-angular-2
AngularJS 2 Yeoman generators (
TypeScript
)https://github.com/kriasoft/react-starter-kit
React.js, Express, Flux, ES6+, JSX, Babel, PostCSS, Webpack, BrowserSync (
Node.js
)https://github.com/yeoman/generator-polymer
Polymer Web Components (
gulp
)
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Programming_languages
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Programming_language_classification
- https://en.wikipedia.org/wiki/Von_Neumann_programming_languages
- https://en.wikipedia.org/wiki/Von_Neumann_architecture#Mitigations
Programming Paradigms
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Programming_paradigms
Lightweight Markup Languages
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Lightweight_markup_languages
BBCode
Homepage: https://www.bbcode.org/
Docs: https://www.bbcode.org/reference.php
Docs: https://www.bbcode.org/examples/
BBCode is a Lightweight markup language
often used by bulletin boards and forums.
Markdown
Homepage: https://daringfireball.net/projects/markdown/
Standard: https://daringfireball.net/projects/markdown/syntax
Docs: https://www.w3.org/community/markdown/wiki/MarkdownImplementations
Docs: https://en.wikipedia.org/wiki/Markdown#Implementations
Docs: https://learnxinyminutes.com/docs/markdown/
Docs: https://guides.github.com/features/mastering-markdown/
Docs: https://help.github.com/articles/github-flavored-markdown/
Docs: https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet
FileExt:
.Md
FileExt:
.md
Markdown is a Lightweight markup language
which can be parsed and transformed to valid HTML-
.
- GitHub and BitBucket support Markdown in Issue Descriptions, Wiki Pages, and Comments
Jupyter Notebook
supports Markdown in Markdown cells
CommonMark
Standard: https://spec.commonmark.org/0.29/
Src: https://github.com/commonmark/commonmark-spec
CommonMark
is one effort to standardize Markdown
.
MyST Markdown
MyST
Src: https://github.com/executablebooks/MyST-NB
Docs: https://myst-parser.readthedocs.io/en/latest/
Docs: https://myst-nb.readthedocs.io/en/latest/
MyST Markdown is CommonMark
Markdown
with support for Sphinx
roles and directives.
jupyter-book
builds HTML, LaTeX, and PDF books from MyST Markdown,ReStructuredText
, andJupyter
notebooks in the sequence listed in _toc.yml.
MediaWiki Markup
Standard: https://www.mediawiki.org/wiki/Markup_spec
Docs: https://en.wikipedia.org/wiki/Help:Wiki_markup#Link_to_another_namespace
Docs: https://www.mediawiki.org/wiki/Help:Formatting
Docs: https://meta.wikimedia.org/wiki/Help:Wikitext_examples
Docs: https://en.wikipedia.org/wiki/Help:Displaying_a_formula
MediaWiki Markup is a Lightweight markup language
"WikiText" which can be parsed and transformed to valid HTML-
.
- Wikipedia is built on MediaWiki, which supports MediaWiki Markup.
RD
Ruby Document Format
Standard: https://github.com/uwabami/rdtool/blob/master/doc/rd-draft.rd
Standard: https://github.com/uwabami/rdtool/blob/master/doc/rd-draft.rd.ja
RD is a Lightweight markup language
for documenting Ruby
code and programs.
Rdoc
Docs: http://docs.seattlerb.org/rdoc/
Docs: https://raw.githubusercontent.com/rdoc/rdoc/master/ExampleRDoc.rdoc
RDoc is a tool and a Lightweight markup language
for generating HTML-
and command-line documentation for Ruby
projects.
To not build RDoc docs when installing a Gem <RubyGems>
:
gem install --no-rdoc --no-ri
gem install --no-document
gem install -N
ReStructuredText
Homepage: http://docutils.sourceforge.net/rst.html
Docs: http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html
Docs: http://docutils.sourceforge.net/docs/ref/rst/directives.html
Docs: http://docutils.sourceforge.net/docs/ref/rst/roles.html
Docs: https://www.sphinx-doc.org/rest.html
ReStructuredText (ReST, RST) is a Lightweight markup language
commonly used for narrative documentation and inline Python, C, Java, etc. docstrings which can be parsed, transformed, and published to valid HTML-
, ePub, LaTeX, PDF.
Sphinx
is built on Docutils
, the primary implementation of ReStructuredText.
Pandoc
also supports a form of ReStructuredText.
- ReStructuredText Directive
Actionable blocks of ReStructuredText
include
,contents
, andindex
are all ReStructuredDirectives:.. include:: goals.rst .. contents:: Table of Contents :depth: 3 .. index:: Example 1 .. index:: Sphinx + .. _example-1: Sphinx +1 ========== This refs :ref:`example 1 <example-1>`. Similarly, an explicit link to this anchor `<#example-1>`__ And an explicit link to this section `<#sphinx-1>`__ (which is otherwise not found in the source text). .. index:: Example 2 .. _example 2: Example 2 ========== This links to :ref:`example-1` and :ref:`example 2`. (`<#example-1>`__, `<#example-2>`__) And this also links to `Example 2`_. .. include:: LICENSE
Note
index
is aSphinx
Directive, which will print an error to the console when building but will otherwise silently dropped by non-Sphinx ReStructuredText parsers likeDocutils
(GitHub) andPandoc
.- ReStructuredText Role
RestructuredText role extensions
:ref:
is aSphinx
RestructuredText Role:A (between files) link to :ref:`example 2`.
C
Docs: https://www.securecoding.cert.org/confluence/display/c/SEI+CERT+C+Coding+Standard
Docs: https://cwe.mitre.org/top25/#CWE-120
Docs: https://cwe.mitre.org/data/definitions/120.html#Demonstrative_Examples
Docs: https://learnxinyminutes.com/docs/c/
C is a third-generation programming language which affords relatively low-level machine access while providing helpful abstractions.
Every Windows
kernel is written in C.
The GNU/Linux
kernel is written in C and often compiled by GCC
or Clang
for a particular architecture (see: man uname
)
The OSX
kernel is written in C.
Libc
libraries are written in C.
Almost all of the projects linked here, at some point, utilize code written in C.
Libc
A libc is a standard library of C
routines.
Libc implementations:
Glibc
BSD Libc <bsd-libc>
- https://en.wikipedia.org/wiki/UClibc
Bionic
GNU Libc
Glibc
Homepage: https://www.gnu.org/software/libc/
Docs: https://www.gnu.org/software/libc/documentation.html
Docs: https://www.gnu.org/software/libc/manual/html_mono/libc.html
Docs: https://sourceware.org/glibc/wiki/HomePage
Src: https://en.wikipedia.org/wiki/GNU_C_Library
Glibc is the GNU C
Library (libc
).
Many Linux
packages and the GNU/Linux <linux>
kernel build from Glibc.
BSD Libc
Src: https://svnweb.freebsd.org/base/head/lib/libc/
Src: https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/
Src: https://opensource.apple.com/source/Libc/
BSD libc are a superset of POSIX
.
OSX
builds from BSD libc.
Android
Bionic
is a BSD libc.
Bionic
Src: git https://github.com/android/platform_bionic
Docs: https://developer.android.com/tools/sdk/ndk/index.html
Bionic is the Android
libc
, which is a BSD Libc
<bsd-libc>
.
C++
C++ is a free and open source third-generation programming language which adds object orientation and a standard library to C
.
- C++ is an ISO specification: C++98, C++03, C++11 (C++0x), C++14, [ C++17 ]
- There are many template libraries for C++: https://en.wikipedia.org/wiki/List_of_C%2B%2B_template_libraries
Standard Template Library
libstdc++
Src: https://github.com/gcc-mirror/gcc/tree/master/libstdc%2B%2B-v3
Docs: https://gcc.gnu.org/onlinedocs/libstdc++/
libstdc++ is the free and open source GNU C++
Standard Template Library
.
GCC
(G++
) typically builds with libstdc++.
Src: https://github.com/llvm/llvm-project/tree/master/libcxx/
Docs: https://libcxx.llvm.org/docs/
libc++ (libcxx) is the free and open source LLVM
C++
Standard Template Library
.
Clang
(clang++
) typically builds with libc++ (libcxx).
Microsoft STL
Microsoft STL is Microsoft's free and open source implementation of the C++
Standard Template Library
.
- Microsoft Visual C++ typically builds with the Microsoft STL.
Boost
Homepage: https://www.boost.org/
Src: https://github.com/boostorg/boost
Docs: https://www.boost.org/doc/
Docs: https://www.boost.org/doc/libs/release/
Docs: https://www.boost.org/doc/libs/release/more/getting_started/
Docs: https://www.boost.org/doc/libs/release/libs/python/doc/html/
Boost is a free and open source set of C++
libraries for doing lots of things in C++.
Fortran
Fortran (or FORTRAN) is a third-generation programming language frequently used for mathematical and scientific computing.
Some of the SciPy
libraries build optimized mathematical Fortran routines.
Haskell
Homepage: https://www.haskell.org/
Download: https://www.haskell.org/downloads
Download: https://www.haskell.org/platform/
Docs: https://www.haskell.org/documentation
Docs: https://learnxinyminutes.com/docs/haskell/
Docs: http://learnyouahaskell.com/chapters
Docs: https://en.wikipedia.org/wiki/Haskell_features
Haskell is a free and open source strongly statically typed purely functional programming language.
Cabal
is the Haskell package manager.
Pandoc
is written in Haskell.
Go
Homepage: https://go.dev/
Src: https://github.com/golang/go
Docs: https://go.dev/doc/
Docs: https://go.dev/doc/effective_go
Docs: https://go.dev/security/fuzz/
Docs: https://pkg.go.dev/
Docs: https://pkg.go.dev/std
Docs: https://pkg.go.dev/testing
LearnXinYMinutes: https://learnxinyminutes.com/docs/go/
Go is a free and open source statically-typed C
-based third generation language.
- Go binaries may be compiled without a libc; may make direct kernel syscalls on the platform or platforms they're compiled for.
- Better Go Playground https://goplay.tools/
Java
Docs: http://javadocs.org/
Docs: https://learnxinyminutes.com/docs/java/
Java is a third-generation programming language which is compiled into code that runs in a virtual machine (JVM
) written in C
for many different operating systems.
JVM
A JVM ("Java Virtual Machine") runs Java
code (classes and JARs).
- There are JVMs available for very many platforms
- Both the JRE and the JDK include a compiled JVM:
- JRE -- Java Runtime Environment (End Users)
- JDK -- Java Developer Kit (Developers)
- Java SE is an implementation specification with things like
java.lang
andjava.io
andjava.net
There are now multiple Java SE Implementations:
Wikipedia: https://en.wikipedia.org/wiki/Java_Platform,_Standard_Edition
https://en.wikipedia.org/wiki/Java_(software_platform)#HistoryOracle Java (was Sun Java)
Wikipedia: https://en.wikipedia.org/wiki/Java_Development_Kit
Download: https://www.oracle.com/technetwork/java/javase/
Download: https://www.oracle.com/technetwork/java/javase/downloads/
Download: https://www.java.com/en/download/
Docs: https://www.java.com/en/download/help/index_installing.xml?os=All+PlatformsOpenJDK (open source)
Wikipedia: https://en.wikipedia.org/wiki/OpenJDK
Homepage: https://openjdk.java.net/
Download: https://openjdk.java.net/install/
Src: https://hg.openjdk.java.net/
Docs: https://wiki.openjdk.java.net/
Docs: https://openjdk.java.net/guide/IcedTea (open source)
Wikipedia: https://en.wikipedia.org/wiki/IcedTea
Java EE ("Java Enterprise Edition") extends Java SE with a number of APIs for web services (
javax.servlet
,javax.transaction
)https://en.wikipedia.org/wiki/Java_Platform,_Enterprise_Edition
Javascript
Docs: https://learnxinyminutes.com/docs/javascript/
Javascript (JS) is a free and open source third-generation programming language designed to run in an interpreter; now specified as ECMAScript
.
All major web browsers support Javascript.
Client-side (web) applications can be written in Javascript.
Server-side (web) applications can be written in Javascript, often with Node.js
, NPM
, and Bower
packages.
Note
Java and JavaScript are two distinctly different languages and developer ecosystems.
ECMAScript
Homepage: https://www.ecma-international.org/ecma-262/
Src: https://github.com/tc39/ecma262#ecmascript
Spec: https://www.ecma-international.org/ecma-262/
Spec: https://tc39.github.io/ecma262/
Spec: https://tc39.es/ecma262/
ECMAScript (ES) is an evolving, formally-specified, weakly-typed scripting language from which Javascript
and ActionScript are derived.
- There are multiple versions of ECMAScript (ES):
- ES1 -- ES1997
- ES2 -- ES1998
- ES3 -- ES1999
- ES5 -- ES2009
- ES6 -- ES2015
- ES7 -- ES2016
- ES8 -- ES2017
- ES9 -- ES2018
- ES10 -- ES2019
- ES.Next
Babel
compiles ECMAScript (ES6+) to Javascript.- Some browsers support various versions (ES7) of ECMAScript.
Firefox
is built upon the SpiderMonkey ECMAScript engine.- Google
Chrome
,Node.JS
, and the latest MicrosoftEdge
are built upon the V8 ECMAEscript engine.
Babel
Homepage: https://babeljs.io/
Src: https://github.com/babel/babel
Docs: https://babeljs.io/docs/en/
Babel is a Javascript
(ECMAScript
) compiler that transforms ES6 (ES2015) and beyond into browser-compatible JS.
ReactJS
developers commonly compile ES6+ andJSX
to JS with Babel.
Node.js
Homepage: https://nodejs.org/
Src: https://github.com/joyent/node
Docs: https://nodejs.org/en/docs/
Docs: https://nodejs.org/api/
Node.js is a free and open source framework for Javascript
applications written in C
, C++
, and Javascript
.
Jinja2
Homepage: https://palletsprojects.com/p/jinja/
Src: https://github.com/pallets/jinja
Docs: https://jinja2.readthedocs.io/en/latest/
Docs: https://jinja.palletsprojects.com/
Jinja2 is a free and open source templating engine written in Python
.
Sphinx
and Salt
are two projects that utilize Jinja2.
Perl
Homepage: https://www.perl.org/
Project: https://dev.perl.org/perl5/
Docs: https://www.perl.org/docs.html
Src: git git://perl5.git.perl.org/perl.git
Perl is a free and open source, dynamically typed, C
-based third-generation programming language.
Many of the Debian
system management tools are or were originally written in Perl.
Python
Homepage: https://www.python.org/
Src: git https://github.com/python/cpython
Src: git https://github.com/python/peps
Docs: https://docs.python.org/2/
Docs: https://docs.python.org/3/
Docs: https://docs.python.org/devguide/
Docs: https://docs.python.org/devguide/documenting.html
Docs: https://wiki.python.org/moin/PythonBooks
Docs: https://www.onlineprogrammingbooks.com/python/
Docs: https://www.reddit.com/r/learnpython/wiki/index
Docs: https://www.reddit.com/r/learnpython/wiki/books
Docs: https://www.quora.com/Python-programming-language-1/What-are-the-best-books-courses-for-learning-Python
Docs: https://en.wikiversity.org/wiki/Python
Docs: https://www.class-central.com/search?q=python
Docs: https://learnxinyminutes.com/docs/python/
Awesome: https://github.com/vinta/awesome-python
Python is a free and open source dynamically-typed, C
-based third-generation programming language.
As a multi-paradigm language with support for functional and object-oriented code, Python is often utilized for system administration and scientific software development.
- Many of the
RedHat
system management tools (e.g. theYum
anddnf
package managers) are written in Python. - The Gentoo
Portage
package manager is written in Python. - The
Conda
package manager is written in Python. IPython
,Pip
,Conda
,Sphinx
,Docutils
,Mercurial
,OpenStack
,Libcloud
,Salt
,Ansible
,Tox
,Virtualenv
, andVirtualenvwrapper
are all written in Python.PyPI
is the Python community index for sharing open sourcepython packages
.Pip
installs from PyPI.
The Python community is generously supported by a number of sponsors and the Python Infrastructure Team:
- https://www.python.org/psf/sponsorship/
- https://www.python.org/psf/members/#sponsor-members
- https://infra.psf.io/
- https://packaging.python.org/
CPython
Homepage: https://www.python.org/
Docs: https://docs.python.org/2/
Docs: https://docs.python.org/devguide/
Docs: https://docs.python.org/devguide/documenting.html
Docs: https://learnxinyminutes.com/docs/python/
Src: hg https://hg.python.org/cpython
CPython is the reference Python
language implementation written in C
.
CPython can interface with other C
libraries through a number of interfaces:
Cython
Homepage: https://cython.org/
PyPI: https://pypi.python.org/pypi/Cython
Docs: https://docs.cython.org/
Docs: https://docs.cython.org/src/userguide/language_basics.html
Cython is a superset of CPython
which adds static type definitions; making CPython
code faster, in many cases.
NumPy
Homepage: https://www.numpy.org/
Src: https://github.com/numpy/numpy
Docs: https://docs.scipy.org/doc/numpy/
NumPy is a library of array-based mathematical functions implemented in C
and Python
.
- https://nbviewer.jupyter.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-2-Numpy.ipynb
- https://scipy-lectures.github.io/intro/numpy/index.html
- https://scipy-lectures.github.io/advanced/advanced_numpy/index.html
NumPy and other languages:
- https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html
- https://github.com/ipython/ipython/wiki/Extensions-Index
SciPy
Homepage: https://scipy.org/
Src: https://github.com/scipy/scipy
Docs: https://www.scipy.org/docs.html
Docs: https://docs.scipy.org/doc/scipy/reference/
Docs: https://www.scipy.org/install.html
SciPy is a set of science and engineering libraries for Python
, primarily written in C
.
- https://nbviewer.jupyter.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-3-Scipy.ipynb
- https://scipy-lectures.github.io/intro/scipy.html
- The
SciPy Stack <scipystack>
specification includes the SciPy package and its dependencies.
SciPy Stack
Python Distributions
Sage Math
(SageMathCloud
)- Enthought Canopy
- Python(x,y)
- WinPython
- Pyzo
Anaconda
(Conda
,Wakari
)
Scipy Stack Docker Containers
Jupyter Docker Stacks
DockerHub: https://hub.docker.com/r/ipython/ipython/
DockerHub: https://hub.docker.com/r/jupyter/
DockerHub: https://hub.docker.com/r/jupyter/datascience-notebook/
DockerHub: https://hub.docker.com/r/jupyter/tmpnb/
Jupyter
andScipystack
Docker
containers:
See also:
PyPy
Homepage: https://pypy.org/
Src: https://bitbucket.org/pypy/pypy
Docs: http://buildbot.pypy.org/waterfall
Docs: https://pypy.readthedocs.io/en/latest/
Docs: https://pypy.readthedocs.io/en/latest/introduction.html
PyPy is a JIT LLVM compiler for Python
code written in RPython -- a restricted subset of CPython
syntax --which compiles to C
, and is often faster than CPython
for many types of purposes.
NumPyPy
NumPyPy is a port of NumPy
to PyPy
:
Docs: http://buildbot.pypy.org/numpy-status/latest.html
Docs: https://pypy.org/numpydonate.html
Python 3
Docs: https://docs.python.org/3/howto/pyporting.html
Docs: https://docs.python.org/3/howto/cporting.html
Docs: https://learnxinyminutes.com/docs/python3/
Python 3 made a number of incompatible changes, requiring developers to update and review their Python 2 code in order to "port to" Python 3.
Python 2 will be supported in "no-new-features" status for quite some time.
Python 3 Wall of Superpowers tracks which popular packages have been ported to support Python 3: https://python3wos.appspot.com/
There are a number of projects which help bridge the gap between the two language versions:
- https://pypi.python.org/pypi/six
- https://six.readthedocs.io/
- https://pypi.python.org/pypi/nine
- https://github.com/nandoflorestan/nine/blob/master/nine/__init__.py
- https://pypi.python.org/pypi/future
- https://python-future.org/
See also: Anaconda
awesome-python-testing
Src: https://github.com/westurner/wiki/blob/master/awesome-python-testing.rest
Tox
Docs: https://tox.readthedocs.io/en/latest/
Src: https://github.com/tox-dev/tox
Pypi: https://pypi.python.org/pypi/tox
Tox is a build automation tool designed to build and test Python projects with multiple language versions and environments in separate virtualenvs <virtualenv>
.
Run the py27 environment:
tox -v -e py27
tox --help
Ruby
Homepage: https://www.ruby-lang.org/
Src: https://github.com/ruby/ruby
Docs: https://www.ruby-lang.org/en/documentation/
Docs: https://learnxinyminutes.com/docs/ruby/
Ruby is a free and open source dynamically-typed programming language.
Vagrant
is written in Ruby.
Rust
Homepage: https://www.rust-lang.org/
Docs: https://doc.rust-lang.org/stable/
Docs: https://doc.rust-lang.org/nightly/
Docs: https://learnxinyminutes.com/docs/rust/
Docs: https://doc.rust-lang.org/book/
Rust is a free and open source strongly typed multi-paradigm programming language.
- It's possible to "drop-in replace"
C
andC++
modules with rust code - Rust can call into
C
code - Rust is similar to but much safer than
C
andC++
("memory safety") In terms of
C
compatibility and smart pointers/references ("memory safety");Go
andRust
have similar objectives.https://doc.rust-lang.org/book/ownership.html
- https://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization
- https://en.wikipedia.org/wiki/Smart_pointer
- https://doc.rust-lang.org/book/the-stack-and-the-heap.html
https://rust-lang.github.io/rlibc/rlibc/fn.memcpy.html
pub unsafe extern fn memcpy(dest: *mut u8, src: *const u8, n: usize) -> *mut u8
Scala
Homepage: https://scala-lang.org/
Src: git https://github.com/scala/scala
Twitter: https://twitter.com/scala_lang
Docs: https://scala-lang.org/api/current/
Docs: https://scala-lang.org/api/current/#scala.collection.mutable.LinkedHashMap
Docs: https://learnxinyminutes.com/docs/scala/
Scala is a free and open source object-oriented and functional programming language
which compiles to JVM
(and LLVM
) bytecode.
TypeScript
Homepage: https://www.typescriptlang.org/
Src: https://github.com/Microsoft/TypeScript
NPM: https://www.npmjs.com/package/typescript
NPMPkg: typescript
Standard: https://github.com/Microsoft/TypeScript/blob/master/doc/spec.md
FileExt:
.ts
Docs: https://www.typescriptlang.org/Tutorial
Docs: https://www.typescriptlang.org/Handbook
Docs: https://learnxinyminutes.com/docs/typescript/
TypeScript is a free and open source Programming Language
developed as a superset of Javascript
with optional additional features like static typing and native object-oriented code.
- Angular 2 is written in
TypeScript
: https://github.com/angular/angular/blob/master/modules/angular2/angular2.ts
WebAssembly
Project: https://www.w3.org/community/webassembly/
Src: https://github.com/WebAssembly
FileExt:
.wast
FileExt:
.wasm
Docs: https://github.com/WebAssembly/design
Docs: https://github.com/WebAssembly/design/blob/master/UseCases.md
WebAssembly (wasm) is a safe (sandboxed), efficient low-level programming language
(abstract syntax tree) and binary format for the web.
- WebAssembly is initially derived from asm.js and PNaCL.
- WebAssembly is an industry-wide effort.
LLVM
can generate WebAssembly from e.g.C
andC++
code.
YAML
Homepage: https://yaml.org
Docs: https://learnxinyminutes.com/docs/yaml/
YAML ("YAML Ain't Markup Language") is a concise data serialization format.
Most Salt
states and pillar data are written in YAML. Here's an example top.sls
file:
base:
'*':
- openssh
'*-webserver':
- webserver
'*-workstation':
- gnome
- i3
Compilers
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Compilers
History of compiler construction
https://en.wikipedia.org/wiki/History_of_compiler_construction
Interpreters
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Interpreters_(computing)
- https://en.wikipedia.org/wiki/List_of_command-line_interpreters
- https://en.wikipedia.org/wiki/List_of_command-line_interpreters#Operating_system_shells
- https://en.wikipedia.org/wiki/List_of_command-line_interpreters#Programming
- https://en.wikipedia.org/wiki/List_of_command-line_interpreters#Programming_languages
- A programming
language
interpreter interprets instructions without an ahead-of-time compilation step. - A programming language interpreter is generally compiled ahead-of-time.
- A programming language interpreter implements a version of a programming language specification.
- Command
Shells
(e.g.Bash
,ZSH
) are programming language interpreters. - Scripting
Languages
(e.g.PERL
,Ruby
,Python
) are interpreted or interpret-able languages. - Interpreted languages can often also be compiled ahead-of-time (e.g. with build optimizations).
Source-to-Source Compiler
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Source-to-source_compilers
ROSE
Homepage: http://rosecompiler.org/
Src: git https://github.com/rose-compiler/rose
Docs: http://rosecompiler.org/?page_id=11
Docs: http://rosecompiler.org/ROSE_Tutorial/ROSE-Tutorial.pdf
Docs: http://www.rosecompiler.org/ROSE_HTML_Reference/
Docs: http://wiki.rosecompiler.org/
Binutils
GNU Binutils
Homepage: https://www.gnu.org/software/binutils/
Src: git git://sourceware.org/git/binutils-gdb.git
Docs: https://sourceware.org/binutils/docs-2.24/
Docs: https://sourceware.org/binutils/docs-2.24/binutils/index.html
Docs: https://sourceware.org/binutils/docs-2.24/as/index.html
Docs: https://sourceware.org/binutils/docs-2.24/ld/index.html
GNU Binutils are a set of utilities for working with assembly and binary.
GCC
utilizes GNU Binutils to compile the GNU/Linux
kernel and userspace.
GAS, the GNU Assembler (as
) assembles ASM code for linking by the GNU linker (ld
).
Clang
Homepage: https://clang.llvm.org/
Src: git https://github.com/llvm/llvm-project
Docs: https://clang.llvm.org/docs/
Docs: https://clang.llvm.org/docs/UsersManual.html
Clang is a compiler front end for C
, C++
, and Objective C/++. Clang is part of the LLVM
project.
GCC
GNU Compiler Collection
Homepage: https://gcc.gnu.org/
Docs: https://gcc.gnu.org/onlinedocs/
Src: git ssh://gcc.gnu.org/git/gcc.git
The GNU Compiler Collection started as a Free and Open Source compiler for C
.
- There are now GCC frontends for many languages, including
C++
,Fortran
,Java
, andGo
. - The
C++
GCC frontend binary is calledg++
.
GNU Linker
ld
The GNU Linker is the GNU implementation of the ld
command for linking object files and libraries.
LLVM
Homepage: https://llvm.org/
Src: git https://github.com/llvm/llvm-project
Docs: https://llvm.org/docs/
Docs: https://llvm.org/docs/GettingStarted.html
Docs: https://llvm.org/docs/ReleaseNotes.html
Docs: https://llvm.org/ProjectsWithLLVM/
LLVM "Low Level Virtual Machine" is a reusable compiler infrastructure with frontends for many languages.
Clang
is an LLVM frontend for C-based languages likeC
,C++
,CUDA
, andOpenCL
.- There is a
WASM
LLVM backend: LLVM can produceWebAssembly
binaries. - The
C++
LLVM frontend binary is calledclang++
.
Operating Systems
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Operating_systems
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Operating_system_technology
History of operating systems
https://en.wikipedia.org/wiki/History_of_operating_systems
Timeline of operating systems
https://en.wikipedia.org/wiki/Timeline_of_operating_systems
List of operating systems
Comparison of operating systems
https://en.wikipedia.org/wiki/Comparison_of_operating_systems
- https://en.wikipedia.org/wiki/List_of_important_publications_in_computer_science#Operating_systems
POSIX
Docs: https://en.wikipedia.org/wiki/POSIX#POSIX-oriented_operating_systems
POSIX ("Portable Operating System Interface") is a set of standards for Shells
, Operating Systems
, and APIs.
GNU/Linux
Linux
Homepage: https://www.kernel.org/
Docs: https://www.kernel.org/doc/
Src: git https://github.com/torvalds/linux
GNU/Linux ("Linux") is a free and open source operating system kernel written in C
.
uname -a; echo "Linux"
uname -o; echo "GNU/Linux"
Linux Distributions
A Linux Distribution is a collection of Packages
compiled to work with a GNU/Linux <linux>
kernel and a libc
.
Arch
Homepage: https://www.archlinux.org/
Download: https://www.archlinux.org/download/
Docs: https://wiki.archlinux.org/
Docs: https://aur.archlinux.org/
Docs: https://aur4.archlinux.org/
Docs: https://wiki.archlinux.org/index.php/Arch_packaging_standards
Docs: https://wiki.archlinux.org/index.php/Arch_User_Repository#AUR_4
Arch Linux is a Linux Distribution <linux-distributions>
that is built from AUR
packages.
Debian
Homepage: https://www.debian.org/
Download: https://www.debian.org/distrib/
DockerHub: https://hub.docker.com/_/debian/
Docs: https://www.debian.org/doc/
Docs: https://www.debian.org/doc/manuals/debian-reference/
Docs: https://www.debian.org/doc/#manuals
Docs: https://www.debian.org/doc/debian-policy/ (main, contrib, non-free)
Docs: https://www.debian.org/releases/stable/releasenotes
Docs: https://www.debian.org/releases/stable/i386/release-notes/
Docs: https://www.debian.org/releases/stable/amd64/release-notes/
Debian is a Linux Distribution <linux-distributions>
that is built from DEB
packages.
Ubuntu
Homepage: https://ubuntu.com/
Src: https://launchpad.net/ubuntu
Src: http://archive.ubuntu.com/
Src: http://releases.ubuntu.com/
Download: https://ubuntu.com/download
DockerHub: https://hub.docker.com/_/ubuntu/
Docs: https://help.ubuntu.com/
Q&A: https://askubuntu.com
Ubuntu is a Linux Distribution <linux-distributions>
that is built from DEB
packages which are often derived from Debian
packages.
Fedora
Homepage: https://getfedora.org/
Download: https://getfedora.org/en/workstation/download/
Download: https://getfedora.org/en/server/download/
Download: https://getfedora.org/en/cloud/download/
DockerHub: https://hub.docker.com/_/fedora/
Docs: https://docs.fedoraproject.org/en-US/index.html
Docs: https://fedoraproject.org/wiki/Fedora_Project_Wiki
Docs: https://fedoraproject.org/wiki/EPEL
Fedora is a Linux Distribution <linux-distributions>
that is built from RPM
packages.
RedHat
RedHat Enterprise Linux
RHEL
Homepage: https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux
Docs: https://access.redhat.com/documentation/en-US/
Docs: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/
RedHat Enterprise Linux ("RHEL") is a Linux Distribution <linux-distributions>
that is built from RPM
packages.
CentOS
Homepage: https://www.centos.org/
Download: https://www.centos.org/download/
Docs: https://wiki.centos.org/
Docs: https://www.centos.org/docs/
DockerHub: https://hub.docker.com/_/centos/
CentOS is a Linux Distribution <linux-distributions>
that is built from RPM
packages which is derived from RHEL <redhat>
.
Scientific Linux
Homepage: https://en.wikipedia.org/wiki/Scientific_Linux
Scientific Linux is a Linux Distribution <linux-distributions>
that is built from RPM
packages which is derived from CentOS
. which is derived from RHEL <redhat>
.
rdfs:seeAlso
Anaconda
(Conda
)rdfs:seeAlso
Portage
Oracle Linux
Oracle Linux is a Linux Distribution <linux-distributions>
that is built from RPM
packages which is derived from RHEL <redhat>
.
Gentoo
Homepage: https://gentoo.org/
Src: https://github.com/gentoo
Src: git https://github.com/gentoo/portage
Docs: https://wiki.gentoo.org/wiki/
Docs: https://wiki.gentoo.org/wiki/Handbook:Main_Page
Docs: https://wiki.gentoo.org/wiki/Handbook:AMD64
Docs: https://wiki.gentoo.org/wiki/Handbook:X86
Docs: https://wiki.gentoo.org/wiki/Project:Portage
Docs: https://wiki.gentoo.org/wiki/Project:Hardened
Gentoo is a Linux Distribution <linux-distributions>
built on Portage
.
- https://hub.docker.com/search?q=gentoo (Stage 3 + Portage)
ChromiumOS
Homepage: https://www.chromium.org/chromium-os
Docs: https://www.chromium.org/chromium-os/quick-start-guide
Docs: https://www.chromium.org/chromium-os/developer-guide
Src: https://chromium.googlesource.com/ (
chromiumos*/
)ChromiumOS is a Linux Distribution <linux-distributions>
built on Portage
.
Crouton
Crouton ("Chromium OS Universal Chroot Environment") installs and debootstraps a Linux Distribution <linux-distributions>
(i.e. Debian
or Ubuntu
) within a ChromiumOS
or ChromeOS
chroot.
ChromeOS
ChromeOS is a Linux Distribution <linux-distributions>
built on ChromiumOS
and Portage
.
- ChromeOS powers Chromebooks
- ChromeOS powers Chromeboxes
CoreOS
Homepage: https://coreos.com/
Src: https://github.com/coreos
Docs: https://coreos.com/docs/
Docs: https://coreos.com/docs/#running-coreos
Docs: https://coreos.com/docs/running-coreos/platforms/vagrant/
Docs: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos.md
CoreOS is a Linux Distribution <linux-distributions>
for highly available distributed computing.
CoreOS schedules redundant docker
images with fleet and systemd according to configuration stored in etcd, a key-value store with a D-Bus interface.
- CoreOS runs on very many platforms
- CoreOS does not provide a package manager
- CoreOS schedules Docker
- CoreOS -- Operating System
- etcd -- Consensus and Discovery
- rkt -- Container Runtime
- fleet -- Distributed init system (etcd, systemd)
- flannel -- Networking
Linux Notes
- https://github.com/westurner/provis
https://github.com/saltstack/salt-bootstrap
curl -L https://bootstrap.saltstack.com scripts/bootstrap-salt.sh
- Masterless Salt Config:
make salt_local_highstate_test
- [ ] Workstation role
- [ ] GRUB chainloader to partition boot record
- Ubuntu and Fedora GRUB try to autodiscover Windows partitions
Android
Homepage: https://www.android.com/
Homepage: https://developer.android.com/
Android SDK
Src: https://android.googlesource.com/
Src: https://github.com/android
Docs: https://developer.android.com/sdk/
Docs: https://developer.android.com/sdk/installing/index.html
Docs: https://developer.android.com/sdk/installing/adding-packages.html
Docs: https://source.android.com/source/index.html
Docs: https://source.android.com/source/downloading.html
Docs: https://source.android.com/source/developing.html
Docs: https://source.android.com/source/contributing.html
Docs: https://sites.google.com/a/android.com/tools/build
Docs: https://developer.android.com/tools/workflow/index.html
Android Studio
Homepage: https://developer.android.com/tools/studio/index.html
Docs: https://developer.android.com/tools/workflow/index.html
Docs: https://sites.google.com/a/android.com/tools/build/studio
Apple OSX
OS X
OSX
Wikipedia: https://en.wikipedia.org/wiki/MacOS
Homepage: https://www.apple.com/osx
Homepage: https://www.apple.com/macos/
Docs: https://developer.apple.com/technologies/mac/
Docs: https://developer.apple.com/library/mac/navigation/
Docs: https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/
Src: https://www.apple.com/opensource/
OS X is a UNIX operating system based upon the Mach kernel from NeXTSTEP, which was partially derived from NetBSD and FreeBSD.
OS X GUI support is built from XFree86/X.org X11
.
OS X maintains forks of many POSIX BSD and GNU tools like bash
, readlink
, and find
.
Homebrew
installs and maintains packages for OS X.
uname; echo "Darwin"
iOS
iOS is a closed source UNIX operating system based upon many components of OSX
adapted for phones and then tablets.
- iOS powers iPhones and iPads
- You must have a Mac with
OSX
and XCode to develop and compile for iOS.
- [ ] Create a fresh install
OSX
USB drive (16GB+) - https://github.com/westurner/dotfiles/blob/master/scripts/
setup_*.sh
- [ ] Manually update to latest versions (of zip, tar.gz, .dmg)
- [ ] Port / wrap
shell <shells>
scripts to / withsalt formulas
and parameters (per-subnet, group, machine, os;salt pillar
):- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_brew.sh #
homebrew
- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_mavericks_python.sh #
python
- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_chrome.sh #
chrome
- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_chromium.sh #
chrome
- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_firefox.sh #
firefox
- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_adobereader.sh # PDF forms, signatures, annotations
- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_vlc.sh (vlc)
- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_f.lux.sh (f.lux, UBY)
- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_powerline_fonts.sh (UBY)
- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_macvim.sh (
vim
) - [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_miniconda.sh (
conda
)
- [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_brew.sh #
- https://github.com/westurner/provis
- [ ] https://github.com/westurner/provis/compare/feature/osx_support
- [ ] create / remap "root" group
[ ] http://docs.saltstack.com/en/latest/topics/installation/osx.html
brew install saltstack
ORpip install salt
- [ ] https://github.com/westurner/provis/compare/feature/osx_support
- [ ] Generate installation media
- [ ] Reboot to recovery partition
- [ ] Adjust partitions
- [ ] Format?
- [ ] Install OS
- [ ] (wait)
- [ ] Manual time/date/language config
- [ ] Run workstation provis scripts
- [ ] Generate / obtain installation media
- [ ] Boot from installation media
- [ ] Manual time/date/language config
- [ ] Run workstation provis scripts
- https://www.howtogeek.com/187410/how-to-install-and-dual-boot-linux-on-a-mac/
- https://www.rodsbooks.com/refind/installing.html#osx
Windows
Homepage: https://www.microsoft.com/en-us/windows/
Docs: https://www.microsoft.com/enable/products/docs/
Docs:
Microsoft Windows is a NT-kernel based operating system.
- There used to be a POSIX compatibility mode.
- Chocolatey maintains a set of
NuGet
packages for Windows.
Windows Subsystem for Linux
WSL
Homepage: https://docs.microsoft.com/en-us/windows/wsl
Source: https://github.com/Microsoft/WSL
Windows Subsystem for Linux (WSL) is a binary compatibility layer which allows many Linux programs to be run on Windows 10+.
The Windows Subsystem for Linux lets developers run a GNU/Linux environment -- including most command-line tools, utilities, and applications -- directly on Windows, unmodified, without the overhead of a virtual machine.
- Windows Subsystem for Linux is not a complete
virtualization
solution; but it does allow you to run e.g.Ubuntu
orFedora
(and thus e.g.Bash
) on a Windows machine. Docker
for Windows is one alternative to Windows Subsystem for Linux.
Windows Sysinternals
Homepage: https://technet.microsoft.com/en-us/sysinternals
Download: https://technet.microsoft.com/en-us/sysinternals/bb842062
Download: https://live.sysinternals.com/
Docs: https://technet.microsoft.com/en-us/sysinternals/bb545027
Docs: https://blogs.technet.microsoft.com/sysinternals/
Windows Sysinternals is a group of tools for working with Windows
.
-
- File & Disk: https://technet.microsoft.com/en-us/sysinternals/bb545046
- Networking: https://technet.microsoft.com/en-us/sysinternals/bb795532
- Process: https://technet.microsoft.com/en-us/sysinternals/bb795533
Process Explorer:
https://technet.microsoft.com/en-us/sysinternals/processexplorer
- You can replace Task Manager (
taskman.exe
) with Process Explorer (procexp.exe
) in many versions of Windows (e.g. so thatctrl-alt-delete
launchesprocexp.exe
Win+R
launches the run dialog## Win+R commands C:/ # open explorer to 'C:/' %UserProfile% # open the user profile env var directory cmd[.exe] # open a command prompt calc # calculator control # control panel services.msc # services management compmgmt.msc # computer management eventvwr.msc # event viewer Wupdmgr # windows update manager
- https://en.wikipedia.org/wiki/PATH_(variable)#DOS.2C_OS.2F2.2C_and_Windows
%PATH%
determines which commands you can run without typing a full absolute path. (Some programs will not run if other programs are not in a directory included in a user's current%PATH%
variable)The
%PATH%
variable is determined by key value pairs in the Windows Registry and can be set for:- The current prompt (with
SET PATH=C:/additionalpath;%PATH%
) - A user
- The machine
https://stackoverflow.com/questions/8358265/how-to-update-path-variable-permanently-from-cmd-windows
- The current prompt (with
- You can replace Task Manager (
- Security: https://technet.microsoft.com/en-us/sysinternals/bb795534
- System Information: https://technet.microsoft.com/en-us/sysinternals/bb795535
- Miscellaneous: https://technet.microsoft.com/en-us/sysinternals/bb842059
WSUS Offline Update
Download: http://download.wsusoffline.net/
Src: svn https://svn.wsusoffline.net/svn/wsusoffline/trunk/
Docs: http://www.wsusoffline.net/docs/
WSUS Offline Update is a free and open source software tool for generating offline Windows
upgrade CDs / DVDs containing the latest upgrades for Windows, Office, and .Net.
- Bandwidth costs: Windows Updates (WSUS) in GB * n_machines (see also: Debtorrent,
Packages
) - "Slipstreaming" an installation ISO is one alternative way to avoid having to spend hours upgrading a factory reinstalled ("reformatted")
Windows
installation
A few annotated excerpts from this Chocolatey NuGet
PowerShell
script https://gist.github.com/westurner/10950476#file-cinst_workstation_minimal-ps1 :
cinst GnuWin
cinst sysinternals # Process Explorer XP
cinst 7zip
cinst curl
- [ ] Install Chocolatey NuGet package manager: http://chocolatey.org
- [ ] Install packages listed here: https://gist.github.com/westurner/10950476
- [ ] (Optional) uncomment salt first (optionally specify master) [OR Install salt]
- [ ] Install salt: https://docs.saltstack.com/en/latest/topics/installation/windows.html
<Win>+R
(Start > Run)- [ ] Run
services.msc
and log/prune unutilized services (e.g. workstation, server) and record changes made
- [ ] Windows MBR chain loads to partition GRUB (Linux)
- [ ] Ubuntu WUBI .exe Linux Installer (XP, 7, 8*)
- It's now better to install to a separate partition from a bootable ISO
- https://help.ubuntu.com/community/UEFI
- Cygwin Windows Linux Userspace: ~ https://chocolatey.org/packages/Cygwin
- https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_windows.c
- http://winappdbg.sourceforge.net/#related-projects
Configuration Management
Wikipedia: https://en.wikipedia.org/wiki/Comparison_of_open-source_configuration_management_software
Ansible
Homepage: https://ansible.com/
Src: https://github.com/ansible/ansible
Docs: https://docs.ansible.com/
Docs: https://docs.ansible.com/ansible/latest/
Docs: https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html
Docs: https://docs.ansible.com/ansible/latest/modules/list_of_all_modules.html
Docs: https://docs.ansible.com/ansible/latest/reference_appendices/galaxy.html
Docs: https://github.com/ansible/molecule
Learnxinyminutes: https://learnxinyminutes.com/docs/ansible/
Ansible is a Configuration Management
tool written in Python
which runs idempotent Ansible Playbooks written in YAML
markup with Jinja2
variables for managing one or more physical and virtual machines running various operating systems over SSH.
Cobbler
Homepage: https://cobbler.github.io/
Download: https://cobbler.github.io/downloads/2.8.x.html
Src: git https://github.com/cobbler/cobbler
Docs: https://cobbler.github.io/manuals/quickstart/
Docs: https://cobbler.github.io/manuals/2.8.0/
Cobbler is a machine image configuration, repository mirroring, and networked booting server with support for DNS, DHCP, TFTP, and PXE.
- Cobbler can template kickstart files for the
RedHat
Anaconda installer - Cobbler can template
Debian
preseed files - Cobbler can PXE boot an ISO over TFTP (and unattended install)
- BusyBox,
SystemRescueCD
,Clonezilla
- BusyBox,
- Cobbler can manage a set of DNS and DHCP entries for physical systems
- Cobbler can batch mirror
RPM
andDEB
repositories (see also: apt-cacher-ng,nginx
) - Cobbler-web is a Django WSGI application; usually configured with
Apache HTTPD
and mod_wsgi.- Cobbler-web delegates very many infrastructure privileges
See also: crowbar, OpenStack
Ironic bare-metal deployment
JuJu
Homepage: https://jaas.ai/
Src: https://github.com/juju/juju
Docs: https://jaas.ai/docs
Docs: https://jaas.ai/search?type=charm
TcpPort: 8001
Juju is a Configuration Management
tool written in Python
which runs Juju Charms written in Python
on one or more systems over SSH, for managing one or more physical and virtual machines running Ubuntu
.
osquery
Src: https://github.com/facebook/osquery
Docs: https://osquery.io/docs/tables/
Docs: https://osquery.readthedocs.io/en/stable/
osquery is a tool for reading and querying many sources of system data with SQL for OSX
and Linux
.
- https://docs.saltstack.com/en/develop/ref/modules/all/salt.modules.osquery.html
- https://github.com/westurner/dotfiles/blob/develop/scripts/osquery-all.sh
Puppet
Homepage: https://puppetlabs.com/
Docs: https://puppet.com/docs
Docs: https://puppet.com/docs/puppet/
Src: git https://github.com/puppetlabs
TcpPort: 8140
Puppet is a Configuration Management
system written in Ruby
which runs Puppet Modules written in Puppet DSL or Ruby
for managing one or more physical and virtual machines running various operating systems.
Salt
Homepage: https://www.saltstack.com
Src: git https://github.com/saltstack/salt
Docs: https://docs.saltstack.com/en/latest/
Docs: https://docs.saltstack.com/en/latest/salt-modindex.html
Docs: https://docs.saltstack.com/en/latest/ref/states/all/index.html
Docs: https://docs.saltstack.com/en/latest/ref/clients/index.html#python-api
Docs: https://docs.saltstack.com/en/latest/topics/development/hacking.html
Docs: https://docs.saltstack.com/en/latest/glossary.html
Pypi: https://pypi.python.org/pypi/salt
Twitter: https://twitter.com/SaltStackInc
IRC:
#salt
TcpPort: 4505 (salt zmq)
TcpPort: 4506 (salt zmq)
TcpPort: 22 (salt-ssh)
Salt is a Configuration Management
system written in Python
which runs Salt Formulas written in YAML
, Jinja2
, Python
for managing one or more physical and virtual machines running various operating systems.
Software Build Tools
Build Automation Tools
Autotools
Homepage: https://www.gnu.org/software/automake/
Homepage: https://www.gnu.org/software/automake/
Docs: https://www.gnu.org/software/autoconf/manual/autoconf.html
Docs: https://www.gnu.org/software/automake/manual/automake.html
Docs: https://www.gnu.org/software/automake/manual/automake.html#Standard-Configuration-Variables
GNU Autotools (GNU Build System) are a set of tools for software build automation: autoconf, automake, libtool, and gnulib.
- The traditional
./configure --help; make; make install
build workflow comes from the GNU Build System. - Autoconf uses a
configure.ac
configure include file in generating aconfigure
script that checks for platform and software dependencies and caches the results in aconfig.status
script, which generatesconfig.h
C header file that caches the results - Automake uses a
Makefile.am
to generate aMakefile.in
makefile include file that generates aGNU Make
Makefile
- GNU Coding Standards define a number of standard configuration variables:
CC
,CFLAGS
,CXX
,CXXFLAGS
,LDFLAGS
,CPPFLAGS
which tools such asGNU Make
automatically add to e.g.GCC
(andld
) build program arguments
$ # autoconf # configure.ac -> ./configure
$ ./configure --help
$ ./configure --prefix=/usr/local --with-this-or-that
make
Bake
Bake is a free and open source software build automation tool similar in form and function to Make
.
- Bake uses
Bakefile
files to describe builds. - Bakefiles can contain e.g.
Bash
andPython
scripts (instead ofMake
syntax)
BUILD
A number of tools use (incompatible) BUILD
files to describe software builds:
- Google
Blaze
Bazel
- Twitter
Pants Build
Google Blaze
Blaze
Blaze is an internal software build automation tool developed by Google.
- Blaze was the first build tool to use
BUILD
files. Bazel
is an open source rewrite of Blaze.
Bazel
Homepage: https://bazel.build/
Src: https://github.com/bazelbuild/bazel
Docs: https://docs.bazel.build/
Docs: https://docs.bazel.build/versions/1.2.0/build-ref.html
Bazel is a free and open source software build automation tool developed as a rewrite of Google Blaze
.
- A
WORKSPACE
orWORKSPACE.bazel
file indicates the root of a Bazel workspace. - Bazel uses
BUILD
(orBUILD.bazel
) files to describe builds. Buck
was released beforeBazel
was open-sourced.
BuckBuild
Buck
Homepage: https://buck.build/
Src: https://github.com/facebook/buck
Docs: https://buck.build/setup/getting_started.html
Docs: https://buck.build/about/overview.html
- Buck uses
BUCK
files to describe builds.
CMake
Homepage: https://cmake.org/
Src: https://gitlab.kitware.com/cmake/cmake
Docs: https://cmake.org/cmake/help/latest/
Docs: https://cmake.org/cmake/help/latest/guide/tutorial/
CMake is a free and open source software build automation tool.
- CMake generates build configurations for a number of tools: Unix Makefiles, Ninja, Visual Studio
Gradle
Homepage: https://gradle.org/
Src: git https://github.com/gradle/gradle
Download: https://gradle.org/downloads
Docs: https://docs.gradle.org/current/release-notes
Docs: https://docs.gradle.org/current/userguide/userguide.html
Twitter: https://twitter.com/gradle
Gradle is a build tool for the Java
JVM
which builds a directed acyclic graph (DAG).
Grunt
Src: git https://github.com/gruntjs/grunt
Docs: https://gruntjs.com/getting-started
Docs: https://gruntjs.com/plugins
Twitter: https://twitter.com/gruntjs
Grunt is a build tool written in Javascript
which builds a directed acyclic graph (DAG).
Gulp
Src: https://github.com/gulpjs/gulp
Docs: https://github.com/gulpjs/gulp/blob/master/docs/
Docs: https://github.com/gulpjs/gulp/blob/master/docs/getting-started.md
Docs: https://gulpjs.com/plugins/
Twitter: https://twitter.com/gulpjs
Gulp is a build tool written in Javascript
which builds a directed acyclic graph (DAG).
Jake
Src: git https://github.com/jakejs/jake
NPMPkg: jake
NPM: https://www.npmjs.com/package/jake
Docs: https://jakejs.com/docs-page.html
Jake is a Javascript
build tool written in Javascript
(for Node.js
) similar to Make
or Rake
.
Make
Homepage: https://www.gnu.org/software/make/
Project: https://savannah.gnu.org/projects/make/
Src: git git://git.savannah.gnu.org/make.git
Docs: https://www.gnu.org/software/make/manual/make.html
GNU Make is a classic, ubiquitous software build automation tool designed for file-based source code compilation which builds a directed acyclic graph (DAG).
Bash
, Python
, and the GNU/Linux
kernel are all built with Make.
Make build task chains are represented in a Makefile
.
Pros
- Simple, easy to read syntax
- Designed to build files on disk (see:
.PHONY
) - Nesting:
make -C <path> <taskname>
- Variable Syntax:
$(VARIABLE_NAME)
or${VARIABLE_NAME}
- Bash completion:
make <tab>
- Python: Initially parseable with disutils.text_file
- Logging: command names and values print to stdout (unless prefixed with
@
)
Cons
- Platform Portability: make is not installed everywhere
- Global Variables: parametrization with shell scripts
VARIABLE_NAME="value" make test
make test VARIABLE_NAME="value"
# ...
export VARIABLE_NAME="value"
make test
Pants
Pants Build
Src: git https://github.com/pantsbuild/pants
Docs: https://pantsbuild.github.io/first_concepts.html
Docs: https://pantsbuild.github.io/install.html
Docs: https://pantsbuild.github.io/first_tutorial.html
Docs: https://pantsbuild.github.io/JVMProjects.html
Docs: https://pantsbuild.github.io/scala.html
Docs: https://pantsbuild.github.io/python-readme.html
Docs: https://pantsbuild.github.io/pex_design.html
Docs: https://pantsbuild.github.io/build_files.html
Docs: https://pantsbuild.github.io/build_dictionary.html
Docs: https://pantsbuild.github.io/options_reference.html
Docs: https://pantsbuild.github.io/export.html
Docs: https://pantsbuild.github.io/internals.html
Docs: https://pantsbuild.github.io/howto_contribute.html
Pants Build is a build tool for JVM
[Java
, Scala
, Android
], C++
, Go
, Haskell
, Node
, and Python
[CPython
] software projects.
- A Pants
BUILD
file defines one or more build targets:- Pants can build Deployable Bundles, Runnable Binaries, Importable Code, Tests, and Generated Code (e.g. Java from
Thrift
.thrift
definitions). - Pants can build
PEX
files (Python EXecutables); which are essentially executable ZIP files with inlined dependency sets. - Pants can build
DEX
files (Android
Dalvik Executables) - https://pantsbuild.github.io/build_files.html
- https://pantsbuild.github.io/build_dictionary.html
- https://pantsbuild.github.io/options_reference.html
- Pants can build Deployable Bundles, Runnable Binaries, Importable Code, Tests, and Generated Code (e.g. Java from
- A Pants
pants.ini
file in a top-level source directory defines options for binaries, tools, goals, tasks (sub-goals) Vim
plugin for Pants BuildBUILD
syntax: https://github.com/pantsbuild/vim-pantsIntelliJ
plugin for Pants Build: https://github.com/pantsbuild/intellij-pants-plugin
Virtualization
Cgroups
Docs: https://www.kernel.org/doc/Documentation/cgroups/
Docs: https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/
Docs: https://docs.fedoraproject.org/en-US/Fedora/17/html-single/Resource_Management_Guide/index.html#sec-How_Control_Groups_Are_Organized
Docs: https://wiki.archlinux.org/index.php/Cgroups
Cgroups are a Linux
mechanism for containerizing groups of processes and resources.
- https://chimeracoder.github.io/docker-without-docker/#1
systemd-nspawn
,systemd-cgroup
machinectl
,systemctl
,journalctl
,
libcontainer
Src: https://github.com/docker/libcontainer
Src: https://github.com/opencontainers/runc/tree/master/libcontainer
libcontainer is a library built by Docker
to replace LXC
.
Libcontainer provides a native
Go
implementation for creating containers with namespaces,cgroups
, capabilities, and filesystem access controls.-- https://github.com/opencontainers/runc/tree/master/libcontainer
- libcontainer is now developed as part of
OCI
runC
.
OCI
Open Container Initiative
Docs: https://www.opencontainers.org/about
Src: https://github.com/opencontainers
Twitter: https://twitter.com/oci_org
The Open Container Initiative (OCI) is a Linux Foundation collaborative project dedicated to developing a working, portable software container specification.
runC
runC is a container abstraction
runc is a CLI tool for spawning and running containers according to the
OCI
specification.
- runC builds upon the
libcontainer
abstraction and theOCI
container specification. - runC works on
Linux
,OSX
, andWindows
. - runC containers do not require a daemon process.
- runC containers can run as e.g. a systemd service unit.
Docker
Homepage: https://www.docker.com/
Src: https://github.com/docker/docker
Docs: https://docs.docker.com/
Awesome: https://github.com/veggiemonk/awesome-docker
Docker is an OS virtualization project written in Go
which utilizes Linux
containers -- first LXC
now libcontainer
/ runC
--to partition process workloads across one or more host systems.
- Dockerfile
A
Dockerfile
contains the instructions needed to create a docker image.- Docker container
A Docker container is an instance of a
Docker Image
with configuration.- Docker API
The Docker API is an interface of management commands for provisioning and managing containers.
Docker Machine
,Docker Swarm
, andDocker Universal Control Plane
all implement the Docker API; so thedocker
client works equally well with each implementation.- Docker Machine
Docker Machine is the container management application which implements the
Docker API
.- Docker Swarm
Docker Swarm is a cluster management system for Docker containers hosted on one or more
Docker Machines <Docker Machine>
- Docker Universal Control Plane
Docker Universal Control Plane is an enterprise-grade cluster management solution with a web dashboard and external authentication which implements the
Docker API
.- Docker Compose
Docker Compose is a Python application for defining and managing services (
Docker containers <docker container>
) and networks with adocker-compose.yml
YAML
configuration file.- Docker Image
A Docker Image is an archived container filesystem with configuration which is usually defined by a
Dockerfile
.- Docker Hub
Docker Hub is a cloud-based registry service for
Docker Images <Docker Image>
.- Docker Cloud
Docker Cloud is the hosting service offered by Docker.
- Docker images build from a
Dockerfile
- A
Dockerfile
can subclass another Dockerfile (to add, remove, or change configuration) Dockerfile
support a limited number of commands- Docker is not intended to be a complete
configuration management system <configuration management>
- Ideally, a Docker image requires minimal configuration once built
- Docker images can be hosted by https://hub.docker.com/
docker run -it ubuntu/16.04
downloads the image from https://hub.docker.com/_/ubuntu/, creates a new instance (docker ps
), and spawns a rootShell <shells>
with a UUID name (by default).- There are a number of ways to "Schedule" [redundant] persistent containers that launch on boot with
Docker
- Docker Swarm is the Docker-native way to run a cluster of containers. To a client app, Docker Swarm looks just like Docker Machine because it implements the Docker API.
Kubernetes
is one project which uses Docker to schedule redundant, optionally geodistributed,LXC
containers (in "Pods").
Salt
can install and manage docker, docker images and containers:
- https://github.com/saltstack-formulas/docker-formula
- https://docs.saltstack.com/en/latest/ref/states/all/salt.states.dockerio.html
- https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.dockerio.html
CNCF
Cloud Native Computing Foundation
The Cloud Native Computing Foundation (CNCF) is a foundation for cloud and container industry collaboration.
Kubernetes
is now a CNCF project.
Kubernetes
Homepage: https://kubernetes.io/
Src: https://github.com/GoogleCloudPlatform/kubernetes
Docs: https://kubernetes.io/gettingstarted/
Docs: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/docker.md
Docs: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/vagrant.md
Docs: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos.md
Docs: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/mesos.md
Q&A: https://stackoverflow.com/questions/tagged/kubernetes
Twitter: https://twitter.com/kubernetesio
Awesome: https://github.com/ramitsurana/awesome-kubernetes
Kubernetes (k8s) is a highly-available distributed cluster scheduler which works with groups of Docker
containers called Pods.
- Google donated Kubernetes to the
CNCF
.
k3s
Src: https://github.com/rancher/k3s
Docs: https://rancher.com/docs/k3s/latest/en/
k3s is a lightweight Kubernetes
distribution which runs on x86-64, ARM6, ARM7; only requires 512Mb of RAM; and is distributed as a single Go
binary.
- You've heard of k8s? This is k8s - 5.
Kubernetes-Mesos
kubernetes-mesos integrates Kubernetes
Docker
Pod scheduling with Mesos
.
Kubernetes and Mesos are a match made in heaven.
Kubernetes enables the Pod, an abstraction that represents a group of co-located containers, along with Labels for service discovery, load-balancing, and replication control.
Mesos provides the fine-grained resource allocations for pods across nodes in a cluster, and facilitates resource sharing among Kubernetes and other frameworks running on the same cluster.
KVM
Homepage: https://www.linux-kvm.org/
Docs: https://www.linux-kvm.org/page/Documents
KVM is a full virtualization platform with support for Intel VT and AMD-V; which supports running various guest operating systems, each with their own kernel, on a given host machine.
Libcloud
Docs: https://libcloud.readthedocs.io/en/latest/
Docs: https://libcloud.readthedocs.io/en/latest/supported_providers.html
Src: git git://git.apache.org/libcloud.git
Src: git https://github.com/apache/libcloud
Apache libcloud is a Python
library which abstracts and unifies a large number of Cloud APIs for Compute Resources, Object Storage, Load Balancing, and DNS.
Salt
salt cloud
depends upon libcloud.
Libvirt
Homepage: https://libvirt.org/
Docs: https://libvirt.org/docs.html
Docs: https://libvirt.org/cgroups.html
Docs: https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.virt.html
Src: git git://libvirt.org/libvirt-appdev-guide.git
Libvirt is a system for platform virtualization with various Linux
hypervisors.
- Xen
- QEMU,
KVM
- OpenVZ,
LXC
VirtualBox
LXC
Homepage: https://linuxcontainers.org/
Docs: https://linuxcontainers.org/lxc/documentation/
Src: https://github.com/lxc/lxc
LXC ("Linux Containers"), written in C
, builds upon Linux
Cgroups
to provide containerized OS chroots (all running under the host kernel <linux>
).
LXC is included in recent Linux
kernels.
LXD
Docs: https://linuxcontainers.org/lxd/
Src: https://github.com/lxc/lxd
LXD, written in Go
, builds upon LXC
to provide a system-wide daemon and an OpenStack
Nova hypervisor plugin.
Mesos
Homepage: https://mesos.apache.org/
Src: git https://github.com/apache/mesos
Q&A: https://stackoverflow.com/tags/mesos
Twitter: https://twitter.com/ApacheMesos
Apache Mesos is a highly-available distributed datacenter operating system, for which there are many different task/process/service schedulers.
Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.
Mesosphere
Src: https://github.com/mesosphere
Q&A: https://stackoverflow.com/tags/mesosphere
Twitter: https://twitter.com/mesosphere
- Apache
Mesos
is a core Mesosphere service
OpenStack
Homepage: https://www.openstack.org/
Src: https://git.openstack.org
Src: https://github.com/openstack
Q&A: https://stackoverflow.com/questions/tagged/openstack
Docs: https://docs.openstack.org/
Docs: https://wiki.openstack.org/
Docs: https://wiki.openstack.org/wiki/Get_OpenStack
Twitter: https://twitter.com/openstack
OpenStack is a platform of infrastructure services for running a cloud datacenter (a private or a public cloud).
- OpenStack can be installed on one machine with enough RAM, or many thousands of machines.
- OpenStack Keystone -- cluster/grid/cloud-level token and user-service based authentication (authN) and authorization (authZ) as a service.
- OpenStack Nova implements a Hypervisor API which abstracts various
Virtualization
providers (e.g.KVM
,Docker
,LXC
,LXD
). - OpenStack Swift -- redundant HTTP-based Object Storage as a service.
- OpenStack Neutron (Quantum)-- software defined networking (SDN), VLAN, switch configuration, virtual and physical enterprise networking as a service.
- OpenStack Designate -- DNS as a service (Bind9, PowerDNS) integrated with OpenStack Keystone, Neutron, and Nova.
- OpenStack Poppy -- CDN as a service CDN vendor API
- OpenStack Horizon -- web-based OpenStack Dashboard which is written in Django.
OpenStack makes it possible for end-users to create a new virtual machine from the available pool of resources.
rdfs:seeAlso
: openstack-devstack
, Libcloud
OpenStack DevStack
Docs: https://docs.openstack.org/devstack/latest/
Docs: https://docs.openstack.org/devstack/latest/overview.html
Issues: https://launchpad.net/devstack
OpenStack DevStack is a default development configuration for OpenStack
.
There are many alternatives to and implementations of OpenStack DevStack:
- https://github.com/saltstack-formulas/openstack-standalone-formula
- https://github.com/CSSCorp/openstack-automation
- https://github.com/openstack-ansible/openstack-ansible
- https://forge.puppetlabs.com/puppetlabs/openstack
- https://jujucharms.com/q/openstack
- https://anvil.readthedocs.io/en/latest/topics/summary.html
Packer
Src: https://github.com/hashicorp/packer
Docs: https://www.packer.io/docs
Docs: https://www.packer.io/docs/basics/terminology.html
Packer generates machine images for multiple platforms, clouds, and hypervisors from a parameterizable template.
- Packer Artifact
Build products: machine image and manifest
- Packer Template
JSON build definitions with optional variables and templating
- Packer Build
Task defined by a JSON file containing build steps which produce a machine image
- Packer Builder
Packer components which produce machine images for one of many platforms:
VirtualBox
Docker
OpenStack
- GCE
- EC2
- VMware
- QEMU (
KVM
, Xen) - https://www.packer.io/docs/templates/builders.html
- Packer Provisioner
Packer components for provisioning machine images at build time
- Shell scripts
- File uploads
- ansible
- chef
- solo
- puppet
- salt
- Packer Post-Processor
Packer components for compressing and uploading built machine images
Vagrant
Homepage: https://www.vagrantup.com/
Docs: https://www.vagrantup.com/docs/
Src: https://github.com/hashicorp/vagrant
Vagrant is a tool written in Ruby
for creating and managing virtual machine instances with CPU, RAM, Storage, and Networking.
- Vagrant:
- Works with a number of Cloud and
Virtualization
providers:VirtualBox
- AWS EC2
- GCE
OpenStack
- provides helpful commandline porcelain on top of
VirtualBox
VboxManage
- installs and lifecycles Vagrant Boxes
- Works with a number of Cloud and
vagrant help
vagrant status
vagrant init ubuntu/trusty64
vagrant up
vagrant ssh
$EDITOR Vagrantfile
vagrant provision
vagrant halt
vagrant destroy
- vagrantfile
vagrant script defining a team of one or more virtual machines and networks.
create a vagrantfile:
vagrant init [basebox] cat vagrantfile
start virtual machines and networks defined in the vagrantfile:
vagrant status vagrant up
- Vagrant Box
Vagrant base machine virtual machine image.
There are many baseboxes for various operating systems.
Essentially a virtual disk plus CPU, RAM, Storage, and Networking metadata.
Locally-stored and cached vagrant boxes can be listed with:
vagrant help box vagrant box list
A running vagrant environment can be packaged into a new box with:
vagrant package
Packer
generatesVirtualBox
Vagrant Boxes with a Post-Processor.- Vagrant Cloud
Vagrant-hosted public Vagrant Box storage.
Install a box from Vagrant cloud:
vagrant init ubuntu/trusty64 vagrant up vagrant ssh
- Vagrant Provider
A driver for running Vagrant Boxes with a hypervisor or in a cloud.
The Vagrant
VirtualBox
Provider is well-supported.With Plugins: https://github.com/mitchellh/vagrant/wiki/Available-Vagrant-Plugins
See also:
libcloud
.- Vagrant Provisioner
Set of hooks to install and run shell scripts and configuration managment tools over
vagrant ssh
.Vagrant up runs
vagrant provision
on first invocation ofvagrant up
.vagrant provision
Note
Vagrant configures a default NFS share mounted at /vagrant
.
Note
Vagrant adds a default NAT Adapter as eth0; presumably for DNS, the default route, and to ensure vagrant ssh
connectivity.
VirtualBox
Homepage: https://www.virtualbox.org/
Docs: https://www.virtualbox.org/wiki/Documentation
Src: svn svn://www.virtualbox.org/svn/vbox/trunk
Oracle VirtualBox is a platform virtualization package for running one or more guest VMs (virtual machines) within a host system.
VirtualBox:
- runs on many platforms:
Linux
, OSX, Windows - has support for full platform NX/AMD-v virtualization
- requires matching kernel modules
Vagrant
scripts VirtualBox.
Shells
Bash
Homepage: https://www.gnu.org/software/bash/
Src: git git://git.savannah.gnu.org/bash.git
Docs: https://www.gnu.org/software/bash/manual/
LearnXinYMinutes: https://learnxinyminutes.com/docs/bash/
Awesome: https://github.com/awesome-lists/awesome-bash
GNU Bash, the Bourne-again shell, is an open source command-line program written in C
for running commands in a text-based terminal.
A few commands to try when learning to shell with Bash:
echo $SHELL; echo "$SHELL"; echo "${SHELL}"
type bash
bash --help
help help
help type
apropos bash
info bash
man bash
man man
info info # [down arrow] and then [enter] to select, or 'n' for next
Bash works with unix command outputs and return codes: a program returns nonzero when there is an error:
true; echo $? # 0 false; echo $? # 1 echo "Hello" && echo " World!" # Hello World! false || echo "World!" # World!
Functions: Bash supports functions with arguments that can print to standard out and/or return an integer return code:
function add_a { echo "$1 + $2 = $(( $1 + $2 ))" } add_b () { echo "$1 + $2 = $(( $1 + $2 ))" } add_xy () { echo "$x + $y = $(( $x + $y ))" } add_a 3 5 # "3 + 5 = 8" add_b 3 5 # "3 + 5 = 8" x=3 y=5 add_xy # "3 + 5 = 8" x=3; y=5; add_xy # "3 + 5 = 8" output=$(add_a 3 5) echo "${output}" help test help [ help [[ help return test "$(add_a 3 5)" == "3 + 5 = 8" && echo 'OK' test_add_a () { if [[ "$(add_a 3 5)" == "3 + 5 = 8" ]]; then echo 'OK' return 0 else echo 'Test failed' return 1 fi } test_add_a help trap help exit
- Portability: sh (sh, bash, dash, zsh) shell scripts are mostly compatible; though bash supports some features that other shells do not.
Logging: You can configure bash to print commands and arguments as bash executes scripts:
set -x # print commands and arguments set -v # print source
Bash reads various configuration files at startup time:
/etc/profile
/etc/bash.bashrc
/etc/profile.d/*.sh
${HOME}/.profile /etc/skel/.profile # PATH=+$HOME/bin # umask
${HOME}/.bash_profile # empty. preempts .profile
${HOME}/.bashrc
Bash and various Operating Systems
:
- Linux: Bash is almost always installed as the default shell on Linux boxes.
- Mac:
- MacOS includes Bash 3.2.
- You can
brew install bash
to get a more recent version. (homebrew
)
- Windows:
Windows Subsystem for Linux
(WSL) installs Linux distributions which include bash.- You can also install bash on Windows by installing git with
choco install git -y
(Chocolatey
) - You can also install bash on Windows by installing MSYS2 (Mingw) or Cygwin with
choco install msys2
orchoco install cygwin
While Bash is ubiquitous, shell scripts are loose with quoting; which makes shell scripts flexible but dangerous and thus often avoided in favor of other languages:
## Shell script quoting example 1:
# This prints a newline
echo $(echo "-e a\nb")
# This prints "-e a\nb"
echo "$(echo "-e a\nb")"
This isn't an issue with e.g. Python
(a popular language that's also useful for system administration).
import subprocess
print(subprocess.check_output(['echo', "-e a\nb"])
print(subprocess.check_output('echo "-e a\nb"', shell=True))
# Though, note that Python subprocess shell=True is a security risk:
# - avoid shell=True
# - pass the command as a list of already-tokenized arguments
# - use something like sarge (or ansible) instead of shell=True
IPython
is one of many alternatives to Bash.
Readline
Homepage: https://tiswww.case.edu/php/chet/readline/rltop.html
Docs: https://tiswww.case.edu/php/chet/readline/readline.html
Docs: https://tiswww.case.edu/php/chet/readline/history.html
Docs: https://tiswww.case.edu/php/chet/readline/rluserman.html
Src: ftp ftp://ftp.gnu.org/gnu/readline/readline-8.0.tar.gz
Pypi: https://pypi.python.org/pypi/gnureadline
IPython
ipython
Homepage: https://ipython.org/
Src: git https://github.com/ipython/ipython
DockerHub: https://hub.docker.com/repos/ipython/
Docs: https://ipython.readthedocs.io/en/stable/
Docs: https://ipython.readthedocs.io/en/stable/interactive/
Docs: https://ipython.readthedocs.io/en/stable/parallel/
Docs: https://github.com/ipython/ipython/wiki/Extensions-Index
Docs: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
Docs: https://github.com/ipython/ipython/wiki/Install:-Docker
IPython is an interactive REPL and distributed computation framework written in Python
.
## Formatting expression output with the Python interpreter
1 + 1
x = 1+1
print("1 + 1 = 2")
print('1 + 1 = %d' % (x))
print('1 + 1 = {0}'.format(x)) # Python 2.7+
print('1 + 1 = {x}'.format(x=x)) # Python 2.7+
print(f'1 + 1 = {x}') # Python 3.6+
print(f'{1 + 1 = }') # Python 3.8+
## IPython
!ipython --help # run `$SHELL -c 'ipython --help'`
!python -m IPython --help # run `ipython --help`
? # print IPython help within IPython
%lsmagic
%<tab> # list magic commands and aliases
%paste? # help for the %paste magic command
%logstart? # help for the %logstart magic command
%logstart -o logoutput.log.py # log input and output to a file
import json
json? # print(json.__doc__)
json?? # print(inspect.getsource(json))
## IPython shell
!cat ./README.rst; echo $PWD # run shell commands
lines = !ls -al # capture shell command output
print(lines[0:])
%run -i -t example.py # run a script with timing info,
# in the local namespace
%run -d example.py # run a script with pdb
%pdb on # automatically run pdb on Exception
- If a kernel is not specified, IPython uses the
ipykernel
Jupyter kernel. To use other kernels with IPython, you must install jupyter_console and a kernel:
pip install jupyter_console # conda install -y jupyter_console ipython console --kernel python # ipykernel jupyter console --kernel python # ipykernel # <Ctrl-D> | <Ctrl-C> | "exit()" pip install bash_kernel # conda install -y bash_kernel jupyter console --kernel bash # "exit" conda install -y -c conda-forge xeus-cling jupyter console --kernel xcpp11 # <Ctrl-D> (<Ctrl-Z> on Windows) conda install -y nodejs; npm install -g ijavascript; ijsinstall jupyter console --kernel javascript # <Ctrl-D> conda install -y nodejs; npm install -g jp-babel; jp-babel-install jupyter console --kernel babel # <Ctrl-D> conda install -y nodejs; npm install -g itypescript; its --install=local jupyter console --kernel typescript # <Ctrl-D> jupyter kernelspec list
- There are very many Jupyter kernels: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
Jupyter Notebook
andJupyter Lab
are built atop IPython: a Jupyter notebook file is aJSON
file with an .ipynb extension which contains inputs and text and binary outputs.
PowerShell
Windows PowerShell
Homepage: https://microsoft.com/powershell
Windows PowerShell is a shell for Windows
.
ZSH
Homepage: https://www.zsh.org/
Docs: http://zsh.sourceforge.net/Guide/zshguide.html
Docs: http://zsh.sourceforge.net/Doc/
Src: git git://git.code.sf.net/p/zsh/code
Shell Utilities
Awk
Docs: https://en.wikipedia.org/wiki/AWK#Versions_and_implementations
GNU Awk
Project: https://savannah.gnu.org/projects/gawk/
Homepage: https://www.gnu.org/software/gawk/
Docs: https://www.gnu.org/software/gawk/manual/
Docs: https://www.gnu.org/software/gawk/manual/gawk.html
Src: git git://git.savannah.gnu.org/gawk.git
AWK is a pattern programming language for matching and transforming text.
Grep
Homepage: https://www.gnu.org/software/grep/
Project: https://savannah.gnu.org/projects/grep/
Docs: https://www.gnu.org/software/grep/manual/
Docs: https://www.gnu.org/software/grep/manual/grep.html
Src: git git://git.savannah.gnu.org/grep.git
Grep is a commandline utility for pattern-based text matching.
Htop
Homepage: https://hisham.hm/htop/
Src: https://github.com/hishamhm/htop
Htop is a commandline task manager; like top
extended.
Pyline
Docs: https://pyline.readthedocs.io/en/latest/
Src: git https://github.com/westurner/pyline
Pypi: https://pypi.python.org/pypi/pyline
Pyline is an open source POSIX
command-line utility for streaming line-based processing in Python
with regex and output transform features similar to grep
, sed
, and awk
.
- Pyline can generate quoted CSV,
JSON <json->
, HTML, etc.
Pyrpo
Src: git https://github.com/westurner/pyrpo
Pypi: https://pypi.python.org/pypi/pyrpo
Pyrpo is an open source POSIX
command-line utility for locating and generating reports from Git
, Mercurial
, Bazaar
, and Subversion
repositories.
Sed
Homepage: https://www.gnu.org/software/sed/
Project: https://savannah.gnu.org/projects/sed
Docs: https://www.gnu.org/software/sed/manual/
Docs: https://www.gnu.org/software/sed/manual/sed.html
Docs: http://sed.sourceforge.net/
Src: git git.savannah.gnu.org/sed.git
GNU Sed is an open source POSIX
command-line utility for transforming text.
Note
BSD Sed
Use <Ctrl-V><tab>
for explicit tabs (as \t
does not work)
Use \\\n
or '$'\n
for newlines (as \n
does not work)
sed -E
should be consistent extended regular expressions between GNU Sed (e.g. Linux) and BSD Sed (FreeBSD, OSX).
OR: brew install gnu-sed
See: https://unix.stackexchange.com/questions/101059/sed-behaves-different-on-freebsd-and-on-linux
See: https://superuser.com/questions/307165/newlines-in-sed-on-mac-os-x
Web Shells
IPython Notebook
IPython notebook
Src: https://github.com/ipython/ipython/tree/3.x/IPython/html
Docs: http://ipython.readthedocs.io/en/stable/notebook/
Docs: http://ipython.readthedocs.io/en/stable/notebook/notebook.html
Docs: http://ipython.readthedocs.io/en/stable/notebook/nbformat.html
Docs: http://ipython.readthedocs.io/en/stable/notebook/nbconvert.html
Docs: http://ipython.readthedocs.io/en/stable/notebook/public_server.html
Docs: http://ipython.readthedocs.io/en/stable/notebook/security.html
Docs: https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks
IPython
Notebook (now Jupyter Notebook
) is an open source web-based shell written in Python
and Javascript
for interactive and literate computing with IPython notebooks composed of raw, markdown, or code input and plaintext- or rich- output cells.
- An IPython notebook (
.ipynb
) is aJSON-
document containing input and output for a linear sequence of cells; which can be exported to many output formats (e.g.HTML-
, RST, LaTeX, PDF); and edited through the web with IPython Notebook. - IPython Notebook is a webapp written on
tornado
, an asynchronous web application framework for Python.- seeAlso: westurner/brw (2007-))
- IPython Notebook supports
Markdown
syntax for comment cells.MyST Markdown
is acommonmark
Markdown
syntax (which can also be used to express entire Jupyter notebooks)
IPython Notebook supports more than 40 different IPython kernels for other languages:
https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages
- IPython Notebook development has now moved to
Jupyter
Notebook; which supports IPython kernels (and defaults to theIPython
CPython
2 <python>
or3 <python3>
kernel).
To start IPython Notebook (assuming the _SRC
variable as defined in Venv
):
pip install ipython[notebook]
# pip install -e git+https://github.com/ipython/ipython@rel-3.2.1#egg=ipython
# https://github.com/ipython/ipython/releases
mkdir $_SRC/notebooks; cd $_SRC/notebooks
ipython notebook
ipython notebook --notebook-dir="${_SRC}/notebooks"
# With HTTPS (TLS/SSL)
ipython notebook \
--ip=127.0.0.1 \
--certfile=mycert.pem \
--keyfile=privkey.pem \
--port=8888 \
--browser=web # (optional) westurner/dotfiles/scripts/web
# List supported options
ipython notebook --help
Warning
IPython Notebook runs code and shell commands as the user the process is running as, on a remote or local machine.
Reproducible SciPy Stack <scipystack>
IPython Notebook / Jupyter Notebook
servers implement best practices like process isolation and privilege separation with e.g. Docker
and/or Jupyter
Hub.
Note
IPython Notebook is now Jupyter Notebook
.
Jupyter Notebook runs Python notebooks with ipykernel
, the IPython
Python
kernel from IPython Notebook
Jupyter kernels
Docs: https://ipython.readthedocs.io/en/stable/install/kernel_install.html
ipykernel
ipython_nose
ipython_nose is an extension for IPython Notebook
(and likely Jupyter Notebook
) for discovering and running test functions starting with test_
(and unittest.TestCase test classes with names containing Test
) with Nose.
- ipython_nose is not (yet?) uploaded to PyPI
- to install ipython_nose from GitHub (with
Pip
andGit
):
pip install -e git+https://github.com/taavi/ipython_nose#egg=ipython_nose
See also:
- TDD: Test Driven Development https://wrdrd.github.io/docs/consulting/software-development#test-driven-development
- Refactoring and Reproducibility
nosebook
nosebook is a tool for finding and running tests in nbformat
IPython Notebooks <ipython notebooks>
and Jupyter Notebooks <jupyter notebook>
with nose
.
See also:
Jupyter
Project Jupyter
Homepage: http://jupyter.org/
Src: https://github.com/jupyter/
DockerHub: https://hub.docker.com/repos/jupyter/
Docs: https://jupyter.readthedocs.io/en/latest/
Awesome: https://github.com/markusschanta/awesome-jupyter
Awesome: https://github.com/quobit/awesome-python-in-education#jupyter
Project Jupyter expands upon components like IPython
and IPython Notebook
to provide a multi-user web-based shell for many languages (Python
, Ruby
, Java
, Haskell
, Julia, R).
IPython |
Jupyter |
|
|
Jupyter Notebook
Src: https://github.com/jupyter/notebook
CondaPkg:
notebook
FileExt:
.ipynb
Docs: https://jupyter-notebook.readthedocs.io/en/stable/
Jupyter
Notebook is an open source shell webapp written in Python
and Javascript
for interactive and literate computing with Jupyter notebooks composed of raw, markdown, or code input and plaintext- or rich- output cells.
.ipynb
files are Jupyter Notebooks saved asJSON-
documents .- An Jupyter notebook is a document containing {meta, input, and output} records for a linear sequence of cells; which can be exported to many output formats (e.g.
HTML-
, RST, LaTeX, PDF, Python,MyST Markdown
); and edited through the web with Jupyter Notebook. - Jupyter Notebook is a webapp written on
tornado
, an asynchronous web application framework for Python.- seeAlso: westurner/brw (2007-))
- Jupyter Notebook supports
Markdown
syntax for comment cells.MyST Markdown
is acommonmark
Markdown
syntax (which can also be used to express entire Jupyter notebooks)
Jupyter Notebook supports more than 40 different Jupyter kernels for other languages:
https://github.com/ipython/ipython/wiki/Jupyter-kernels-for-other-languages
To start IPython Notebook (assuming the _SRC
variable as defined in Venv
):
pip install ipython[notebook]
# pip install -e git+https://github.com/ipython/ipython@rel-3.2.1#egg=ipython
# https://github.com/ipython/ipython/releases
mkdir $_SRC/notebooks; cd $_SRC/notebooks
ipython notebook
ipython notebook --notebook-dir="${_SRC}/notebooks"
# With HTTPS (TLS/SSL)
ipython notebook \
--ip=127.0.0.1 \
--certfile=mycert.pem \
--keyfile=privkey.pem \
--port=8888 \
--browser=web # (optional) westurner/dotfiles/scripts/web
# List supported options
ipython notebook --help
Warning
IPython Notebook runs code and shell commands as the user the process is running as, on a remote or local machine.
Reproducible SciPy Stack <scipystack>
IPython Notebook / Jupyter Notebook
servers implement best practices like process isolation and privilege separation with e.g. Docker
and/or Jupyter
Hub.
Note
JupyterLab
(a mostly-rewrite) adds e.g. tabs and undo (and a new extension API) to Jupyter Notebook
.
JupyterLab
Src: https://github.com/jupyterlab/jupyterlab
CondaPkg:
jupyterlab
Docs: https://jupyterlab.readthedocs.io/en/stable/
Docs: https://jupyterlab.readthedocs.io/en/stable/user/extensions.html
Docs: https://jupyterlab.readthedocs.io/en/stable/developer/extension_dev.html
Awesome: https://github.com/mauhai/awesome-jupyterlab
JupyterLab is an open-source web-based tabbed IDE <IDE>
written in Python
, Javascript
, and TypeScript
for working with <jupyter notebooks> Jupyter Notebook
, terminals, text editing, undo, extensions.
- You can edit Jupyter notebooks with JupyterLab.
- Installing JupyterLab also installs
Jupyter Notebook
. (which doesn't support tabs or the new extension API) - A few UI differences between JupyterLab and Jupyter Notebook:
- JupyterLab has tabbed editing: you can open files, notebooks, and terminals in tabs
- JupyterLab has a sidebar with a file selector pane
- Installing JupyterLab does not install any
scipystack
or other packages.
JupyterLab (install)
Install JupyterLab With pip
:
python -m pip install jupyterlab
Install JupyterLab with conda
:
conda install -c conda-forge -y jupyterlab
Hosting JupyterLab
JupyterLab (hosting)
You can host JupyterLab yourself:
JupyterHub
- "Zero to JupyterHub with
Kubernetes
" https://zero-to-jupyterhub.readthedocs.io/en/latest/ - JupyterHub Spawners create new instances of JupyterLab (within containers, VMs, a shell)
- JupyterHub Authenticators check names and credentials from a file, PAM, Single Sign On APIs
- "Zero to JupyterHub with
BinderHub
runs containers withrepo2docker
- https://binderhub.readthedocs.io/en/latest/zero-to-binderhub/
mybinder.org
is powered byBinderHub
Hosted JupyterLab
JupyterLab (hosted)
There are many providers of hosted JupyterLab
and Jupyter Notebook
; where they run Jupyter in a shell or a VM on their servers for you and you connect over your internet connection.
- https://github.com/markusschanta/awesome-jupyter#hosted-notebook-solutions
Cocalc
Google AI Platform
has hosted JupyterLab Notebooks.Google Colab
is a fork ofJupyter Notebook
.GitHub Codespaces
(VSCode
)ml-workspace
-- https://github.com/ml-tooling/ml-workspace (VSCode
)MyBinder.org
JupyterHub
Docs: https://jupyterhub.readthedocs.io/en/latest/
Docs: https://github.com/jupyter/jupyterhub/wiki
Docs: https://github.com/jupyter/jupyterhub/wiki/Authenticators
Docs: https://github.com/jupyter/jupyterhub/wiki/Spawners
JupyterHub makes it easy to serve Jupyter Notebook
and/or Jupyter Lab
for multiple users on one or more servers.
- JupyterHub spawns individual Jupyter Notebook / JupyterLab server instances for logged-in users.
- JupyterHub enables users to log-in with Authenticator backends: system users, LDAP, SSO, OAuth (e.g. Google accounts)
- If so configured, JupyterHub can launch additional servers to serve one or more Notebook/Lab
Docker
containers and then shut those down when they're idle or, for example, when a course session is complete.
nbconvert
nbconvert is the code that converts (transforms) an .ipynb
notebook (nbformat
JSON <json->
) file into an output representation (e.g. HTML
, HTML slides (reveal.js
), LaTeX
, PDF, ePub, Mobi).
nbconvert is included with
Jupyter Notebook
andJupyterLab
pip install nbconvert # pip install -e git+https://github.com/jupyter/nbconvert@master#egg=nbconvert jupyter nbconvert --to html mynotebook.ipynb
reveal.js
reveal.js is a Javascript
and HTML
library for slide presentations served from an HTML file.
- Reveal.js slides can be in a 1-dimensional or a 2-dimensional arrangement.
You can generate reveal.js slides from Jupyter notebooks in two ways: with
nbconvert --to slides
or with the GUI: "File" > "Export Notebok As..." > "Export Notebook to reveal.js slides"jupyter nbconvert --to slides mynotebook.ipynb
Note
Presentation content that doesn't fit on a slide is hidden and unscrollable: only put a slide worth of data in each cell for a Jupyter reveal.js presentation.
Alternatives to presenting notebooks as reveal.js slides:
- Increase the browser font size (Jupyter Notebook)
- "View" > "Presentation Mode" (JupyterLab)
- Select a keyboard shortcut set use the "Select Cell Below" / "Select Cell Above" keyboard shortcuts to highlight cells and scroll them into view
- Press "<Escape>"
- Press "j" to "Select Cell Below"
- Press "k" to "Select Cell Above"
- The
RISE
extension also generates reveal.js slides.
RISE
RISE is a Jupyter Notebook
and JupyterLab
extension that generates live reveal.js
presentations from Jupyter notebooks.
- Install the RISE extension
- Click the RISE button to generate a live
reveal.js
slide presentation wherein you can execute cells on the slides with "Ctrl-Enter" and "Shift-Enter" just like you can in the Notebook interface.
nbformat
Docs: https://nbformat.readthedocs.io/en/latest/
Docs: https://nbformat.readthedocs.io/en/latest/format_description.html#backward-compatible-changes
The Jupyter notebook
(.ipynb
) format is a versioned JSON <json->
format for storing metadata and input/output sequences.
Usually, when the nbformat changes, notebooks are silently upgraded to the new version on the next save.
Note
nbformat v3 and above add a kernelspec attribute to the nbformat JSON <json->
, because .ipynb
files can now contain code for languages other than Python
.
- nbformat does not specify any schema for the user-supplied metadata dict (TODO:
nbmeta
), so JSON that conforms to an externally managedJSON-LD <json-ld->
@context
would work.
nbgrader
Docs: https://nbgrader.readthedocs.io/en/stable/
Docs: https://nbgrader.readthedocs.io/en/stable/user_guide/
Docs: https://nbgrader.readthedocs.io/en/latest/
Docs: https://nbgrader.readthedocs.io/en/latest/user_guide/
Docs: https://nbgrader.readthedocs.io/en/latest/command_line_tools/
nbgrader is a solution for centrally receiving and grading Jupyter
notebooks <jupyter notebook>
.
- You mark notebook cells as TODO
See also:
CoCalc
Course management- https://wrdrd.github.io/docs/consulting/education-technology#jupyter-and-tdd
jupyter and tdd
nbviewer
Src: git https://github.com/jupyter/nbviewer
Dockerfile: https://github.com/jupyter/nbviewer/blob/master/Dockerfile
(nbviewer
) is an application for serving read-only versions of Jupyter notebooks from HTTP URLs.
- When you enter a URL, GitHub
username
, GitHubusername/repo
, or Gist ID into the text box at https://nbviewer.jupyter.org/ and click 'Go!' (or press Enter), nbviewer nbconverts the notebook to HTML or shows a file browser and branch/tag selector for the git repo. - You do not need to look up the raw GitHub URL for the notebook, because nbviewer automatically rewrites the GitHub /blob/ file URL to a raw.githubusercontent.com URL.
- GitHub now also renders static
.ipynb
files, CSV, SVG, and PDF. However, GitHub does not execute any JS in the notebook due to security concerns (XSS) - GitLab renders Jupyter notebooks with JS.
runipy
runipy runs Jupyter notebooks <jupyter notebook>
from a Shell <shells>
commandline, generates HTML reports, and can write errors to stderr.
Jupyter notebook <Jupyter Notebook>
manual test review process:
# - run Jupyter Notebook server
!jupyter notebook
# - Browser
# - navigate to / upload / drag and drop the notebook
!web http://localhost:8888 # or https://
# - (optional) click 'TODO Restart Kernel'
# - (optional) click 'Cell' > 'All Output' > 'Clear'
# - click 'Cell' > 'Run All'
# - [wait] <Jupyter Kernel runs notebook>
# - visually seek for the first ERRoring cell (scroll)
# - review the notebook
for (i, o) in notebook_cells:
human.manually_review((i, o))
# - Compare the files on disk with the most recent commit (HEAD)
!git status && git diff
!git diff mynotebook.ipynb
# - Commit the changes
!git-add-commit "TST: mynotebook: tests for #123" ./mynotebook.ipynb
Jupyter notebook <Jupyter Notebook>
TODO review process:
# - run Jupyter Notebook server
!jupyter notebook
# - Browser
# - navigate to / upload / drag and drop the notebook
!web http://localhost:8888 # or https://
# - (optional) click 'TODO Restart Kernel'
# - (optional) click 'Cell' > 'All Output' > 'Clear'
# - click 'Cell' > 'Run All'
# - [wait] <Jupyter Kernel runs notebook>
# - visually seek for the first ERRoring cell (scroll)
# - review the notebook
for (i, o) in notebook_cells:
human.manually_review((i, o))
# - Compare the files on disk with the most recent commit (HEAD)
!git status && git diff
!git diff mynotebook.ipynb
# - Commit the changes
!git-add-commit "TST: mynotebook: tests for #123" ./mynotebook.ipynb
Jupyter notebook <Jupyter Notebook>
runipy review process:
# - runipy the Jupyter notebook
!runipy mynotebook.ipynb
# - review stdout and stderr from runipy
# - review in browser (optional; recommended)
# - navigate to the converted HTML
!web ./mynotebook.ipynb.html
# - visually seek for the first WEEoring cell (scroll)
# - review the notebook
for (i, o) in notebook_cells:
human.manually_review((i, o))
# - Compare the files on disk with the most recent commit (HEAD)
!git status && git diff
!git diff mynotebook.ipynb*
# - Commit the changes
!git-add-commit "TST: mynotebook: tests for #123" ./mynotebook.ipynb*
- An example of runipy usage in a
Makefile <Make>
: https://github.com/westurner/notebooks/blob/gh-pages/Makefile
jupyter_contrib_nbextensions
Docs: https://github.com/ipython-contrib/jupyter_contrib_nbextensions#1-install-the-python-package
CondaPkg:
jupyter_contrib_nbextensions
- https://github.com/ipython-contrib/jupyter_contrib_nbextensions#pip
- https://github.com/ipython-contrib/jupyter_contrib_nbextensions#conda
NBPresent
Docs: https://docs.continuum.io/anaconda/jupyter-notebook-extensions#notebook-present
remix your
Jupyter Notebooks <jupyter notebook>
as interactive slideshows
Anaconda Jupyter Notebook Extensions
Docs: https://docs.continuum.io/anaconda/jupyter-notebook-extensions
Conda
environments,Anaconda
,Jupyter Notebook
CoCalc
SageMathCloud
Homepage: https://cocalc.com/
Src: https://github.com/sagemathinc/cocalc
Src: https://github.com/sagemathinc/cocalc-docker
ChromeExt: https://chrome.google.com/webstore/detail/cocalc/eocdndagganmilahaiclppjigemcinmb
Docs: https://doc.cocalc.com/
Docs: https://doc.cocalc.com/teaching-instructors.html
Docs: https://doc.cocalc.com/teaching-students.html
- https://cocalc.com/
- Interactive Worksheets
- Course Management: Assignments, Handouts,
nbgrader
LaTeX
EditorJupyter Notebook
- Linux Terminal
Sage Math
Google Colab
Colab
Homepage: https://colab.research.google.com/
Src: https://github.com/googlecolab
Docs: https://research.google.com/colaboratory/faq.html
Awesome: https://github.com/firmai/awesome-google-colab
Google Colab is a hosted Jupyter Notebook
system.
- Colab has a number of packages installed in the default environment. If you want additional packages, you need to
!pip install
them once when you first open the notebook. - Colab is forked from a previous version of Jupyter Notebook, and so does not have some newer Jupyter Notebook or any Jupyter Lab features.
ipywidgets
are not yet implemented on Colab.- Colab saves to Google Drive.
- Colab instances are free and can use some GPU time if needed.
- There is a Colab Pro.
- Google AI Platform Notebooks hosts
JupyterLab
notebooks: https://cloud.google.com/ai-platform-notebooks
Dotfiles
Dotfiles are userspace shell <shells>
configuration in files that are often prefixed with "dot" (e.g. ~/.bashrc
for Bash
)
- https://dotfiles.github.io/
westurner/dotfiles
Src: git https://github.com/westurner/dotfiles
Src: https://github.com/westurner/dotfiles/blob/master/etc/.bashrc
Docs: https://westurner.github.io/dotfiles/
Docs: https://westurner.github.io/dotfiles/usage#bashFeatures
- [x] Linear
etc/bash
andetc/zsh
(/etc/bash
) load sequence - [x]
make
Makefile
to log the whole load sequence and grep out docs - [x]
HubFlow
git
branches - [x]
venv
,virtualenv
,virtualenvwrapper
- [x] oh-my-zsh
- [-] bash-it
- [x] Linear
Dotvim
Dotvim is a conjunction / contraction of Dotfiles
and Vim
(in reference to a ~/.vim/
directory and/or a ~/.vimrc
).
Src: https://github.com/westurner/dotvim/blob/master/vimrc
Src: https://github.com/westurner/dotvim/blob/master/vimrc.full.bundles.vimrc
Src: https://github.com/westurner/dotvim/blob/master/vimrc.tinyvim.bundles.vimrc
Docs: https://westurner.github.io/dotfiles/usage#vim
Venv
Docs: https://westurner.github.io/dotfiles/dotfiles.venv
Src: https://github.com/westurner/dotfiles/blob/develop/src/dotfiles/venv/
Src: https://github.com/westurner/dotfiles/blob/develop/etc/bash/10-bashrc.venv.sh
Venv is a tool for making working with Virtualenv
, Virtualenvwrapper
, Bash
, ZSH
, Vim
, and IPython
within a project context very easy.
Venv defines standard fhs
and Python
paths, environment variables, and aliases for routinizing workflow.
var name | description | cdaliases Bash: IPython: Vim: |
example path |
HOME |
user home directory | Bash/ZSH: IPython: Vim: |
~/ |
__WRK |
workspace root | cdwrk (ibid.) |
~/-wrk |
WORKON_HOME |
virtualenvs root | cdwh , cdworkonhome , cdve |
~/-wrk/-ve27 |
CONDA_ENVS_PATH |
condaenvs root | cdch , cdcondahome |
~/-wrk/-ce27 |
VIRTUAL_ENV |
virtualenv root | cdv , cdvirtualenv |
~/-wrk/-ve27/dotfiles |
_BIN |
virtualenv executables | cdb , cdbin |
~/-wrk/-ve27/dotfiles/bin |
_ETC |
virtualenv configuration | cd , cdetc |
~/-wrk/-ve27/dotfiles/etc |
_LIB |
virtualenv lib directory | cdl , cdlib |
~/-wrk/-ve27/dotfiles/lib |
_LOG |
virtualenv log directory | cdlog |
~/-wrk/-ve27/dotfiles/var/log |
_SRC |
virtualenv source repositories | cds , cdsrc |
~/-wrk/-ve27/dotfiles/src |
_WRD |
virtualenv working directory | cdw , cdwrd |
~/-wrk/-ve27/dotfiles/src/dotfiles |
To generate this venv config:
python -m dotfiles.venv.ipython_config --print-bash dotfiles
venv.py --print-bash dotfiles
venv --print-bash dotfiles docs
venv --print-bash dotfiles ~/path
venv --print-bash ~/-wrk/-ve27/dotfiles ~/path
To generate a default venv config with a prefix of /
:
venv --print-bash --prefix=/
To launch an interactive shell within a venv:
venv --run-bash dotfiles
venv -xb dotfiles
Note
pyvenv
is the Virtualenv
-like functionality now included in Python >= 3.3 <python3>
(python3 -m venv
)
Python pyvenv docs: https://docs.python.org/3/library/venv.html
Virtualenv
Docs: https://virtualenv.pypa.io/en/latest/
Src: git https://github.com/pypa/virtualenv
PyPI: https://pypi.python.org/pypi/virtualenv
IRC: #pip
Virtualenv is a tool for creating reproducible Python
environments.
Virtualenv sets the shell environment variable $VIRTUAL_ENV
when active.
Virtualenv installs a copy of Python
, Setuptools
, and Pip
when a new virtualenv is created.
A virtualenv is activated by source
-ing ${VIRTUAL_ENV}/bin/activate
.
Paths within a virtualenv are more-or-less FHS <fhs>
standard paths, which makes virtualenv structure very useful for building chroot and container overlays.
A standard virtual environment:
bin/ # pip, easy_install, console_scripts
bin/activate # source bin/activate to work on a virtualenv
include/ # (symlinks to) dev headers (python-dev/python-devel)
lib/ # libraries
lib/python2.7/distutils/
lib/python2.7/site-packages/ # pip and easy_installed packages
local/ # symlinks to bin, include, and lib
src/ # editable requirements (source repositories)
# also useful
etc/ # configuration
var/log # logs
var/run # sockets, PID files
tmp/ # mkstemp temporary files with permission bits
srv/ # local data
Virtualenvwrapper
wraps virtualenv.
echo $PATH; echo $VIRTUAL_ENV
python -m site; pip list
virtualenv example # mkvirtualenv example
source ./example/bin/activate # workon example
echo $PATH; echo $VIRTUAL_ENV
python -m site; pip list
ls -altr $VIRTUAL_ENV/lib/python*/site-packages/** # lssitepackages -altr
Note
Venv
extends virtualenv
and virtualenvwrapper
.
Note
Python 3.3+ now also contain a script called venv, which performs the same functions and works similarly to virtualenv: https://docs.python.org/3/library/venv.html.
Virtualenvwrapper
Src: hg https://bitbucket.org/dhellmann/virtualenvwrapper
PyPI: https://pypi.python.org/pypi/virtualenvwrapper
Virtualenvwrapper is a tool which extends virtualenvwrapper.
Virtualenvwrapper provides a number of useful shell commands and python functions for working with and within virtualenvs <virtualenv>
, as well as project event scripts (e.g. postactivate
, postmkvirtualenv
) and two filesystem configuration variables useful for structuring development projects of any language within virtualenvs <virtualenv>
: $PROJECT_HOME
and $WORKON_HOME
.
Virtualenvwrapper is sourced into the shell:
# pip install --user --upgrade virtualenvwrapper
source ~/.local/bin/virtualenvwrapper.sh
# sudo apt-get install virtualenvwrapper
source /etc/bash_completion.d/virtualenvwrapper
Note
Venv
extends virtualenv
and virtualenvwrapper
.
echo $PROJECT_HOME; echo ~/workspace # venv: ~/-wrk
cd $PROJECT_HOME # venv: cdp; cdph
echo $WORKON_HOME; echo ~/.virtualenvs # venv: ~/-wrk/-ve27
cd $WORKON_HOME # venv: cdwh; cdwrk
mkvirtualenv example
workon example # venv: we example
cdvirtualenv; cd $VIRTUAL_ENV # venv: cdv
echo $VIRTUAL_ENV; echo ~/.virtualenvs/example # venv: ~/-wrk/-ve27/example
mkdir src ; cd src/ # venv: cds; cd $_SRC
pip install -e git+https://github.com/westurner/dotfiles#egg=dotfiles
cd src/dotfiles; cd $VIRTUAL_ENV/src/dotfiles # venv: cdw; cds dotfiles
head README.rst
# venv: cdpylib
cdsitepackages # venv: cdpysite
lssitepackages
deactivate
rmvirtualenv example
lsvirtualenvs; ls -d $WORKON_HOME # venv: lsve; lsve 'ls -d'
Window Managers
Docs: https://wiki.archlinux.org/index.php/Window_manager
Compiz
Homepage: https://launchpad.net/compiz
Docs: http://wiki.compiz.org/
Src: bzr branch lp:compiz
Compiz is a window compositing layer for X11
which adds lots of cool and productivity-enhancing visual capabilities.
Compiz works with Gnome
, KDE
, and Qt
applications.
f.lux
Download: https://justgetflux.com/dlmac.html
Download: https://justgetflux.com/dlwin.html
Src: git https://github.com/Kilian/f.lux-indicator-applet
Docs: https://justgetflux.com/linux.html
Docs: https://justgetflux.com/ios.html
Docs: https://justgetflux.com/research.html
f.lux is a userspace utility for gradually adjusting the blue color channel throughout the day; or as needed.
A similar effect can be accomplished with the
X11
xgamma
command (e.g. forLinux
platforms where the latest f.lux is not yet available). A few keybindings from ani3wm
configuration here:# [...] #L105 set $xgamma_reset xgamma -gamma 1.0 set $xgamma_soft xgamma -bgamma 0.6 -ggamma 0.9 -rgamma 0.9 set $xgamma_soft_red xgamma -bgamma 0.4 -ggamma 0.6 -rgamma 0.9 # [...] #L200 ## Start, stop, and reset xflux # <alt> [ -- start xflux bindsym $mod+bracketleft exec --no-startup-id $xflux_start # <alt> ] -- stop xflux bindsym $mod+bracketright exec --no-startup-id $xflux_stop # <alt><shift> ] -- reset gamma to 1.0 bindsym $mod+Shift+bracketright exec --no-startup-id $xgamma_reset # <alt><shift> [ -- xgamma -bgamma 0.6 -ggamma 0.9 -rgamma 0.9 bindsym $mod+Shift+bracketleft exec --no-startup-id $xgamma_soft # <alt><shift> \ -- xgamma -bgamma -0.4 -ggamma 0.4 -rgamma 0.9 bindsym $mod+Shift+p exec --no-startup-id $xgamma_soft_red
Gnome
Homepage: https://www.gnome.org/
Docs: https://help.gnome.org/
Src: https://git.gnome.org/browse/
i3wm
Homepage: https://i3wm.org/
Download: https://i3wm.org/downloads/
Docs: https://i3wm.org/docs/
Src: git git://code.i3wm.org/i3
i3wm is a tiling window manager for X11
(Linux
) with extremely-configurable Vim
-like keyboard shortcuts.
i3wm works with Gnome
, KDE
, and Qt
applications.
- An example open source i3wm
i3/config
dotfile <dotfiles>
: https://github.com/westurner/dotfiles/blob/master/etc/i3/config
KDE
Homepage: https://www.kde.org/
Docs: https://docs.kde.org/
Docs: https://www.kde.org/documentation/
Src: https://techbase.kde.org/Getting_Started/Sources
Src: https://techbase.kde.org/Getting_Started/Sources/Subversion
Src: https://techbase.kde.org/Development/Git
Src: https://projects.kde.org/projects
KDE is a GUI framework built on Qt
.
KWin is the main KDE window manager for X11
.
Qt
Homepage: https://qt-project.org/
Homepage: https://www.qt.io/
Docs: https://doc.qt.io/
Docs: https://doc.qt.io/qt-5/qtexamplesandtutorials.html
Docs: https://www.qt.io/contribute/
Docs: https://wiki.qt.io/Main_Page
Docs: https://wiki.qt.io/Get_the_Source
Src: git https://code.qt.io/cgit/
Qt is a Graphical User Interface toolkit for developing applications with Android, iOS, OSX
, Windows, Embedded Linux
, and X11
.
Wayland
Homepage: https://wayland.freedesktop.org/
Src: https://gitlab.freedesktop.org/wayland/wayland
Wayland is a display server protocol for GUI window management.
Wayland is an alternative to X11
servers like XFree86 and X.org.
The reference Wayland implementation, Weston, is written in C
.
X Window System
X11
Homepage: https://www.x.org/
Docs: https://www.x.org/wiki/Documentation/
Src: git git://anongit.freedesktop.org/git/xorg/
X Window System (X, X11) is a display server protocol for window management (drawing windows on the screen).
Most UNIX and Linux
systems utilize XFree86 or the newer X.org X11 window managers.
Gnome
, KDE
, I3wm
, OSX
, and Compiz
build upon X11.
Browsers
Blink
Homepage: https://www.chromium.org/blink
Src: https://src.chromium.org/viewvc/blink/trunk/
Src: git https://chromium.googlesource.com/chromium/blink/
Docs: https://www.chromium.org/blink#TOC-Subpage-Listing
Docs: https://www.chromium.org/blink/developer-faq
Blink is a web browser <browsers>
layout engine written in C++
which was forked from WebKit
.
- Blink now powers
Chrome
andChromium
(Desktop, Mobile),Opera
, Amazon Silk,Android
WebView 4.4+, andQt
WebEngine
Chrome
Homepage: https://www.google.com/chrome/
Download: https://www.google.com/chrome/browser/desktop/
Download: https://www.google.com/chrome/browser/mobile/
AndroidApp: https://play.google.com/store/apps/details?id=com.android.chrome
AndroidAppBeta: https://play.google.com/store/apps/details?id=com.chrome.beta
iOSApp: https://itunes.apple.com/us/app/chrome-web-browser-by-google/id535886823
Docs: https://developer.chrome.com
Docs: https://developer.chrome.com/extensions/devguide
Google Chrome is a Web Browser built from the open source Chromium
browser.
- Google Chrome is now based on
Blink
. - Google Chrome was based on
WebKit
. - Google Chrome includes and updates Adobe Flash, pdf.js
See also: ChromeOS
.
Chromium
Homepage: https://www.chromium.org/Home
Src: git https://chromium.googlesource.com/chromium/src
Src: https://chromium.googlesource.com/chromium/src.git
Docs: https://www.chromium.org/developers
Docs: https://www.chromium.org/developers/coding-style
The Chromium Projects include the Chromium
Browser and chromiumos
.
- Chromium Projects are written in
C++11 <c++>
,C
, IDL,Jinja2
,Python
,Javascript
,HTML-
, andCSS-
. Chrome
andChromeOS
build fromChromium
project sources.
Chrome DevTools
How to open Chrome (and Firefox) DevTools:
- Right-click > "Inspect Element"
- Linux:
<ctrl><shift>i
- OSX:
<option><command>i
DevTools Emulation
- Resize Window to iPhone, iPad, Nexus, Galaxy (landscape / portrait)
- Emulates touch events
- https://developers.google.com/web/fundamentals/tools/devices/browseremulation?hl=en
Chrome Extensions
Accessibility
- Accessibility Developer Tools
- ChromeVox
- Deluminate
- High Contrast
- Spectrum
- Stylish
- Tiësto (Dark Chrome Theme)
- See also:
f.lux
Safety
Content
Tab
Development
- AngularJS Batarang
- FireBug (see: Chrome DevTools)
- FireLogger for Chrome
- JSONView
- ng-inspector for AngularJS
- Postman
- React Developer Tools
- Responsive Web Design Tester
- Requirify
- Web Developer
- Window Resizer
Vim
pbm
PyPI: https://pypi.python.org/pypi/pbm
Warehouse: https://warehouse.python.org/project/pbm
- backup and organize {
Chrome
,Chromium
}Bookmarks
JSON <JSON->
in an offline batch - date-based transforms
- quicklinks
- starred bookmarks (with trailing
##
)
Chrome Android
Extensions
Chrome Android does not support extensions.
Wandroid
- https://bitbucket.org/westurner/wandroid/src/tip/wandroid/apps/chrome/config.py
- https://bitbucket.org/westurner/wandroid/src/tip/wandroid/apps/chrome/userdata.py
Firefox
Firefox Extensions
Accessibility
Safety
Content
Tabs
Development
Vim
Firefox Android
Firefox Android Extensions
Internet Explorer
Internet Explorer is the web browser included with Windows
.
See also: Microsoft Edge
Microsoft Edge
Microsoft Edge will be replacing Internet Explorer
.
Opera
Homepage: https://www.opera.com/
Download: https://www.opera.com/computer/windows
Download: https://www.opera.com/computer/mac
Download: https://www.opera.com/computer/linux
Download: https://www.opera.com/computer/beta
Download: https://www.opera.com/mobile
AndroidApp: https://play.google.com/store/apps/details?id=com.opera.browser
AndroidApp: https://play.google.com/store/apps/details?id=com.opera.mini.native
AndroidApp: https://play.google.com/store/apps/details?id=com.opera.max.global #Proxy Compression
iOSApp: https://itunes.apple.com/app/id363729560 #Opera Mini
iOSApp: https://itunes.apple.com/app/id674024845 #Opera Coast
Src: https://github.com/operasoftware
Opera is a multi-platform web browser <browsers>
written in C++
.
- Opera is now based on
Blink
. - Opera was based on
WebKit
. - Opera developed and open sourced celery: a distributed task worker composed workflow process API written in
Python
; with support for many message browsers: https://github.com/celery
Safari
Homepage: https://developer.apple.com/safari/
Src:
Docs: https://developer.apple.com/safari/resources/
Docs: https://developer.apple.com/library/safari/navigation/
Docs: https://developer.apple.com/library/safari/documentation/Tools/Conceptual/SafariExtensionGuide/Introduction/Introduction.html
Docs: https://developer.apple.com/library/safari/documentation/AppleApplications/Conceptual/Safari_Developer_Guide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40007874
Safari is the web browser included with OSX
.
- Safari is derived from and supports
WebKit
Safari Extensions
Safety
Content
Development
Safari iOS
WebKit
Homepage: https://www.webkit.org/
Download: https://webkit.org/build-archives/
Src: https://github.com/WebKit/webkit
Docs: https://www.webkit.org/coding/coding-style.html
Docs: https://trac.webkit.org/wiki
WebKit is an open source web browser <browsers>
written in C++
.
- WebKit powers
Safari
Browser Extensions
Browser Extensions > Accessibility
- Google Chrome Accesibility Extensions: https://chrome.google.com/webstore/category/ext/22-accessibility?hl=en
- Mozilla Firefox Accesibility Extensions: https://addons.mozilla.org/en-US/firefox/search/?q=accessibility
Accessibility Developer Tools
ChromeVox
ChromeExt: https://chrome.google.com/webstore/detail/chromevox/kgejglhpjiefppelpmljglcjbhoiplfn
Dark Reader
Homepage: https://darkreader.org/
ChromeExt: https://chrome.google.com/webstore/detail/dark-reader/eimadpbcbfnmbkopoojfekhnkhdbieeh
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/darkreader/
SafariExt: https://darkreader.org/safari
EdgeExt: https://microsoftedge.microsoft.com/addons/detail/ifoakfbpdcdoeenechcleahebpibofpc
Twitter: https://twitter.com/darkreaderapp
Deluminate
Src: https://github.com/abstiles/deluminate
ChromeExt: https://chrome.google.com/webstore/detail/deluminate/iebboopaeangfpceklajfohhbpkkfiaa
GitHub Dark Theme
Src: https://github.com/poychang/github-dark-theme
ChromeExt: https://chrome.google.com/webstore/detail/github-dark-theme/odkdlljoangmamjilkamahebpkgpeacp
FirefoxExt: https://addons.mozilla.org/addon/github-dark-theme/
High Contrast
NASA Night Launch
FirefoxXPI: https://addons.mozilla.org/firefox/downloads/latest/4908/addon-4908-latest.xpi
ShadowFox
Homepage: https://overdodactyl.github.io/ShadowFox/
Docs: https://github.com/overdodactyl/ShadowFox/wiki
Spectrum
ChromeExt: https://chrome.google.com/webstore/detail/spectrum/ofclemegkcmilinpcimpjkfhjfgmhieb
Stylish
FirefoxExt: https://addons.mozilla.org/en-us/firefox/addon/stylish/
FirefoxXPI: https://addons.mozilla.org/firefox/downloads/latest/2108/addon-2108-latest.xpi
Docs: https://userstyles.org/help/
Docs: https://userstyles.org/help/stylish_chrome
Docs: https://userstyles.org/help/stylish_firefox
Tiësto
The Tiësto Chrome Theme is a Dark Theme for Chrome.
Browser Extensions > Safety
Safety Extensions
Facebook Container
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/facebook-container/
Firefox Multi-Account Containers
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/multi-account-containers/
Ghostery
Src: https://www.ghostery.com/en/download
FirefoxExt: https://addons.mozilla.org/en-us/firefox/addon/ghostery/
FirefoxXPI: https://addons.mozilla.org/firefox/downloads/latest/9609/addon-9609-latest.xpi
ChromeExt: https://chrome.google.com/webstore/detail/ghostery/mlomiejdfkolichcflejclcbmpeaniij
OperaExt: https://addons.opera.com/addons/extensions/details/ghostery/
SafariExt: https://www.ghostery.com/safari/Ghostery.safariextz
MSIEExt: https://www.ghostery.com/ie/ghostery-ie.exe
AndroidApp: https://play.google.com/store/apps/details?id=com.ghostery.android.ghostery
iOSApp: https://itunes.apple.com/us/app/ghostery/id472789016
FirefoxAndroidXPI: https://addons.mozilla.org/android/downloads/latest/ghostery
FirefoxAndroidExt: https://addons.mozilla.org/en-US/android/addon/ghostery/
HTTPS Everywhere
Homepage: https://www.eff.org/https-everywhere
Src: https://github.com/EFForg/https-everywhere
ChromeExt: https://chrome.google.com/webstore/detail/gcbommkclmclpchllfjekcdonpmejbdp
FirefoxXPI: https://www.eff.org/files/https-everywhere-latest.xpi
FirefoxAndroidXPI: https://www.eff.org/files/https-everywhere-android.xpi
Twitter: https://twitter.com/HTTPSEverywhere
HTTPS Everywhere is a browser extension that forces the browser to only connect over HTTPS to sites listed in its database.
uBlock
Src: https://github.com/chrisaljoudi/ublock
Download: https://github.com/chrisaljoudi/uBlock/releases/latest
ChromeExt: https://chrome.google.com/webstore/detail/ublock/epcnnfbjfcgphgdmggkamkmgojdagdnn
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/ublock/
SafariExt: https://extensions.apple.com/details/?id=net.gorhill.uBlock-96G4BAKDQ9
Docs: https://www.ublock.org/faq/
_repo="chrisaljoudi/ublock"
curl -Ls "https://api.github.com/repos/${_repo}/releases" > ./releases.json
cat releases.json \
| grep browser_download_url \
| pyline 'w and w[1][1:-1]' \
| pyline --regex \
'.*download/(.*)/(uBlock.(firefox.xpi|chromium.zip))$' \
'rgx and rgx.group(1,2)'
uBlock Origin
ChromeExt: https://chrome.google.com/webstore/detail/ublock-origin/cjpalhdlnbpafiamejdnhcphjbkeiagm
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/
EdgeExt: https://microsoftedge.microsoft.com/addons/detail/odfafepnkmbhccpbejgmiehpchacaeak
SafariExt:
Browser Extensions > Content
Content Extensions
Hypothesis
Src: https://github.com/hypothesis/h
ChromeExt: https://chrome.google.com/webstore/detail/hypothesis-web-pdf-annota/bjfhmglciegochdpefhhlphglcehbmek
Twitter: https://twitter.com/hypothes_is
Hypothesis can also be included as a sidebar on a site:
<script async defer src="//hypothes.is/embed.js"></script>
OpenLink Structured Data Sniffer
Homepage: http://osds.openlinksw.com/
ChromeExt: https://chrome.google.com/webstore/detail/openlink-structured-data/egdaiaihbdoiibopledjahjaihbmjhdj?hl=en
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/openlink-structured-data-sniff/
EdgeExtDoc: http://osds.openlinksw.com/#DownloadEdge
JSON-LD
,Microdata
,RDFa
,Turtle
ChromeApp: https://chrome.google.com/webstore/detail/pocket/mjcnijlhddpbdemagnpefmlkjdagkogk
ChromeExt: https://chrome.google.com/webstore/detail/save-to-pocket/niloccemoadcdkdjlinkgdfekeahmflj
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/read-it-later/
FirefoxXPI: https://addons.mozilla.org/firefox/downloads/latest/7661/addon-7661-latest.xpi
SafariLink: https://getpocket.com/safari/
iOSLink: https://getpocket.com/ios/
Zotero
ChromeExt: https://chrome.google.com/webstore/detail/zotero-connector/ekhagklcjbdpajgpjgmbionohlpdbjgc
FirefoxXPI: https://download.zotero.org/extension/zotero-4.0.26.4.xpi
SafariExt: https://download.zotero.org/connector/safari/Zotero_Connector-4.0.21-1.safariextz
Download: https://www.zotero.org/download/
Docs: https://www.zotero.org/support/
Docs: https://www.zotero.org/support/sync
Docs: https://www.zotero.org/support/kb/webdav_services
Zotero archives and tags resources with bibliographic metadata.
- Zotero is really helpful for research.
- Browsers other than Firefox connect to Zotero Standalone
- Zotero can store a full-page archive of a given resource (e.g. HTML, PDF)
- Zotero can store and synchronize data on Zotero's servers with Zotero File Storage
- Zotero can store and synchronize data over WebDAV
- Zotero can export a collection of resources' bibliographic metadata in one of many citation styles ("CSL") (e.g. MLA, APA, [Journal XYZ])
- Zotero can export a collection of resources' bibliographic metadata as RDF
There are a number of plugins and integrations with Zotero:
Zotero and Schema.org RDFa
> How would I go about adding HTML + RDFa [1] and/or HTML + Microdata [2] export templates with Schema.org classes and properties to Zotero?
- https://groups.google.com/forum/#!topic/zotero-dev/rJnMZYrhwM4
- https://lists.w3.org/Archives/Public/public-vocabs/2014Apr/0202.html (COinS, Citeproc-js, OpenAnnotation (+1))
Browser Extensions > Tabs
Tab Extensions
better-onetab
ChromeExt: https://chrome.google.com/webstore/detail/better-onetab/eookhngofldnbnidjlbkeecljkfpmfpg
FoxyTab
Support: https://github.com/erosman/support/issues
OneTab
ChromeExt: https://chrome.google.com/webstore/detail/onetab/chphlpgkkbolifaimnlloiipkdnihall
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/onetab/
FirefoxXPI: https://addons.mozilla.org/firefox/downloads/latest/525044/addon-525044-latest.xpi
- https://github.com/Greduan/chrome-ext-tabulator
better-onetab
Snipe
Src: https://github.com/josephschmitt/Snipe
ChromeExt: https://chrome.google.com/webstore/detail/snipe/glmjakogmemenallddiiajdgjfoogegl
Tabs Outliner
Tree Style Tab
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/tree-style-tab/
Browser Extensions > Development
Development Extensions
AngularJS Batarang
ChromeExt: https://chrome.google.com/webstore/detail/angularjs-batarang-stable/niopocochgahfkiccpjmmpchncjoapek
FireBug
FirefoxExt: https://addons.mozilla.org/en-us/firefox/addon/firebug/
FirefoxXPI: https://addons.mozilla.org/firefox/downloads/latest/1843/addon-1843-latest.xpi
ChromeExt: https://chrome.google.com/extensions/detail/bmagokdooijbeehmkpknfglimnifench
FireLogger
Src: https://github.com/binaryage/firelogger
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/firelogger/
FirefoxXPI: https://addons.mozilla.org/firefox/downloads/file/226289/firelogger-1.3-fx.xpi
Docs: https://github.com/binaryage/firelogger/wiki
- Python: https://github.com/binaryage/firelogger.py
- PHP: https://github.com/binaryage/firelogger.php
- ColdFusion: http://cffirelogger.riaforge.org/
- Java: https://github.com/clescot/webappender
FireLogger for Chrome
ChromeExt: https://chrome.google.com/webstore/detail/firelogger-for-chrome/hmagilfopmdjkeomnjpchokglfdfjfeh
See: FireLogger
JSONView
ChromeExt: https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc
ng-inspector for AngularJS
AngularJS ng-inspector
Src: https://github.com/rev087/ng-inspector
ChromeExt: https://chrome.google.com/webstore/detail/ng-inspector-for-angularj/aadgmnobpdmgmigaicncghmmoeflnamj
FirefoxXPI: http://ng-inspector.org/ng-inspector.xpi
SafariExt: http://ng-inspector.org/ng-inspector.safariextz
Postman
ChromeExt: https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm
Twitter: https://twitter.com/postmanclient
Preact Devtools
Homepage: https://preactjs.github.io/preact-devtools/
ChromeExt: https://chrome.google.com/webstore/detail/preact-developer-tools/ilcajpmogmhpliinlbcdebhbcanbghmd
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/preact-devtools/
EdgeExt: https://microsoftedge.microsoft.com/addons/detail/hdkhobcafnfejjieimdkmjaiihkjpmhk
React Developer Tools
Refined Github
ChromeExt: https://chrome.google.com/webstore/detail/refined-github/hlepfoohegkhhmjieoechaddaejaokhf
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/refined-github-/
Responsive Web Design Tester
Homepage: https://www.esolutions.se/
See also: Chrome DevTools <chrome-devtools>
Emulation
Requirify
Src: https://github.com/mathisonian/requirify
ChromeExt: https://chrome.google.com/webstore/detail/requirify/gajpkncnknlljkhblhllcnnfjpbcmebm
NPM: https://www.npmjs.com/package/requirify
Docs: https://github.com/jfhbrook/browserify-cdn
Requirify adds NPM modules to the local namespace (e.g. from Chrome DevTools JS console).
> require() npm modules in the browser console
require('jquery');
require('d3');
local-requirify
Require local NPM modules with Requirify
Web Developer Extension
Homepage: https://chrispederick.com/work/web-developer/
Src: https://github.com/chrispederick/web-developer/
FirefoxExt: https://addons.mozilla.org/en-us/firefox/addon/web-developer/
FirefoxXPI: https://addons.mozilla.org/firefox/downloads/latest/60/addon-60-latest.xpi
Web Developer Extension, originally just for Firefox, adds many useful developer tools and bookmarklets in a structured menu.
Window Resizer
See also: Chrome DevTools <chrome-devtools>
Emulation
Vim Extensions
Vimium
Homepage: https://vimium.github.io/
Src: git https://github.com/philc/vimium
ChromeExt: https://chrome.google.com/webstore/detail/vimium/dbepggeogbaibhgnhhndojpepiihcmeb
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/vimium-ff/
SafariExt: https://github.com/televator-apps/vimari
Vimium is a Chrome Extension which adds vim
-like functionality.
function | vimium shortcut |
|
? |
|
f / F |
|
yf |
|
p / P |
|
Vimperator
Homepage: http://www.vimperator.org/
Src: https://github.com/vimperator/vimperator-labs
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/vimperator/
Vimperator connects a JS shell with VIM command interpretation to the Firefox API, with vim
-like functionality.
vimperatorrc
can configure settings inabout:config
- Vimperator stopped working after Firefox 57
Wasavi
Src: https://github.com/akahuku/wasavi
ChromeExt: https://chrome.google.com/webstore/detail/wasavi/dgogifpkoilgiofhhhodbodcfgomelhe
OperaExt: https://addons.opera.com/en/extensions/details/wasavi/
FirefoxExt: https://addons.mozilla.org/en-US/firefox/addon/wasavi/
Docs: https://appsweets.net/wasavi/
Wasavi converts the focused textarea
to an in-page editor with vim
-like functionality.
Web Servers
ACME
LetsEncrypt
Homepage: https://letsencrypt.org/
Src: https://github.com/letsencrypt
Src: https://github.com/certbot/certbot
Docs: https://letsencrypt.org/docs/
Docs: https://letsencrypt.org/docs/ct-logs/
- List of LetEncrypt ACME clients: https://letsencrypt.org/docs/client-options/
Apache HTTPD
Homepage: https://httpd.apache.org/
Download: https://httpd.apache.org/download.cgi
Docs: https://httpd.apache.org/docs/2.4/
Apache HTTPD is a scriptable, industry-mainstay HTTP server written in C
and C++
.
BusyBox HTTPD
busybox httpd --help
busybox httpd -p 8082
See also: python http.server
Caddy
Src: https://github.com/caddyserver/caddy
Docs: https://caddyserver.com/docs/
Netcat
# Serve a file over HTTP then close
{ printf 'HTTP/1.0 200 OK\r\nContent-Length: %d\r\n\r\n' "$(wc -c < some.file)"; cat some.file; } | nc -l 8082
# Serve the date over HTTP then start another server
while true ; do nc -l -p 8082 -c 'echo -e "HTTP/1.1 200 OK\n\n $(date -Is)"'; done &
# Make an HTTP request with netcat
printf "GET / HTTP/1.0\r\nHost: localhost\r\n\r\n" | nc localhost 8082
# Make an HTTP request with curl
curl localhost:8082
curl -v localhost:8082
# Make an HTTP request with wget
wget -O - localhost:8082
wget -d -O - localhost:8082
from urllib.request import urlopen
resp = urlopen("http://localhost:8082")
assert resp.code == 200
assert resp.headers.get_content_type() == 'text/plain'
body = resp.read()
print(body)
ncat
nc --help
ncat --help
Nginx
Homepage: https://nginx.org/
Download: https://nginx.org/en/download.html
Src: https://github.com/nginx/nginx
Docs: https://nginx.org/en/docs/
Twitter: https://twitter.com/nginxorg
Nginx is a scriptable, lightweight HTTP server written in C
.
Python http.server
Docs: https://docs.python.org/3/library/http.server.html
Docs: https://docs.python.org/3/library/http.server.html#http.server.CGIHTTPRequestHandler
python -m http.server --help
python -m http.server --directory . 8082
python -m http.server --directory . --cgi 8082
See also: pgs
Tengine
Homepage: https://tengine.taobao.org/
Src: git https://github.com/alibaba/tengine
Download: https://tengine.taobao.org/download.html
Docs: https://tengine.taobao.org/documentation.html
Tengine is a fork of Nginx
with many useful modules and features bundled in.
- https://tengine.taobao.org/document/http_ssl.html
- https://tengine.taobao.org/document/http_upstream_check.html
- https://tengine.taobao.org/document/http_reqstat.html
Traefik
Homepage: https://traefik.io/traefik/
Docs: https://doc.traefik.io/traefik/
Docs: https://doc.traefik.io/traefik/https/acme/
Docs: https://doc.traefik.io/traefik/user-guides/docker-compose/acme-tls/
Docs: https://doc.traefik.io/traefik/reference/dynamic-configuration/docker/
- Load Balancing, API Gateway (config, stats),
Kubernetes Ingress
,ACME
/LetsEncrypt
Kubernetes Ingress
Docs: https://kubernetes.io/docs/concepts/services-networking/ingress/
Docs: https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
Documentation
Docutils
PyPI: https://pypi.python.org/pypi/docutils
Docs: http://docutils.sourceforge.net/docs/
Docs: http://docutils.sourceforge.net/rst.html
Docs: http://docutils.sourceforge.net/docs/ref/doctree.html
Docs: https://docutils.readthedocs.io/en/sphinx-docs/
Docs: https://docutils.readthedocs.io/en/sphinx-docs/ref/rst/restructuredtext.html
Src: svn http://svn.code.sf.net/p/docutils/code/trunk
Docutils is a Python
library which 'parses" ReStructuredText
lightweight markup language into a doctree (~DOM) which can be serialized into HTML, ePub, MOBI, LaTeX, man pages, Open Document files, XML, JSON, and a number of other formats.
Pandoc
Homepage: https://pandoc.org/
Docs: https://pandoc.org/README.html
Docs: https://pandoc.org/releases.html
Download: https://github.com/jgm/pandoc/releases
BrewPkg: pandoc
AptPkg: pandoc
YumPkg: pandoc
Pandoc is a "universal" markup converter written in Haskell
which can convert between HTML, BBCode
, Markdown
, MediaWiki Markup <mediawiki markup>
, ReStructuredText
, HTML, and a number of other formats.
pgs
pgs is an open source web application written in Python
for serving static files from a git
branch, or from the local filesystem.
pgs -p "${_WRD}/_build/html" -r gh-pages -H localhost -P 8082
- pgs is written with the one-file Bottle web framework
- compared to
python -m SimpleHTTPServer localhost:8000
/python3 -m http.server localhost:8000
pgs has WSGI, the ability to read from a Git branch without realgit
bindings, and caching HTTP headers based on Git or filesystem mtimes. - pgs does something like
Nginx
try_files $.html
https://westurner.github.io/tools/index
https://westurner.github.io/tools/index.html
Sphinx
can also generate links without.html
extensions with thehtml_link_suffix
conf.py
configuration setting.https://github.com/westurner/tools/blob/master/conf.py :
# Suffix for generated links to HTML files. # The default is whatever html_file_suffix is set to; # it can be set differently (e.g. to support different web server setups). html_link_suffix = ''
- Many web analytics tools support rules for deduplicating
<name>.html
and<name>
(which GitHub Pages always supports).
Sphinx
Homepage: https://pypi.python.org/pypi/Sphinx
Src: git https://github.com/sphinx-doc/sphinx
Pypi: https://pypi.python.org/pypi/Sphinx
Docs: https://www.sphinx-doc.org/contents.html
Docs: https://www.sphinx-doc.org/markup/code.html
Docs: https://www.sphinx-doc.org/en/stable/markup/inline.html#ref-role
Docs: https://pygments.org/docs/lexers/
Docs: https://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html
Docs: https://github.com/yoloseem/awesome-sphinxdoc
Sphinx is a tool for working with ReStructuredText
documentation trees and rendering them into HTML, PDF, LaTeX, ePub, and a number of other formats.
Sphinx extends Docutils
with a number of useful markup behaviors which are not supported by other ReStructuredText parsers.
Most other ReStructuredText parsers do not support Sphinx directives; so, for example,
GitHub and BitBucket do not support Sphinx but do support ReStructuredText so
README.rst
containing Sphinx tags renders in plaintext or raises errors.For example, the index page of this
Sphinx
documentation set is generated from a file namedindex.rst
that referenced bydocs/conf.py
, which is utilized bysphinx-build
in theMakefile
.Input:
_indexrst="$WORKON_HOME/src/westurner/tools/index.rst" e $_indexrst # with westurner/dotfiles.venv mkvirtualenv westurner we westurner tools; mkdir -p $_SRC git clone ssh://git@github.com/westurner/tools cdw; e index.rst # ew index.rst
https://github.com/westurner/tools/blob/master/index.rst
https://raw.githubusercontent.com/westurner/tools/master/index.rst
Output:
cd $_WRD # cdwrd; cdw git status; make <tab> # gitw status; makew <tab> make html singlehtml # make docs web ./_build/html/index.html # make open make gh-pages # ghp-import -n -p ./_build/html/ -b gh-pages make push # gitw push <origin> <destbranch>
Output: ReadTheDocs:
https://<projectname>.readthedocs.io/en/<version>/
- Sphinx Builder
A Sphinx Builder transforms
ReStructuredText
into various output forms:- HTML
- LaTeX
- ePub
- MOBI
- JSON
- OpenDocument (OpenOffice)
- Office Open XML (MS Word)
See: Sphinx Builders
- Sphinx ReStructuredText
Sphinx extends
ReStructuredText
with roles and directives which only work with Sphinx.- Sphinx Directive
Sphinx extensions of
Docutils
ReStructuredText
directives.Most other ReStructuredText parsers do not support Sphinx directives.
.. toctree:: readme installation usage
See: Sphinx Directives
- Sphinx Role
Sphinx extensions of
Docutils
RestructuredText
rolesMost other ReStructured
.. _anchor-name: A link to :ref:`anchor <anchor-name>`.
jupyter-book
MyST Markdown
(Sphinx roles and directives inMarkdown
)
nbsphinx
Tinkerer
Src: https://github.com/vladris/tinkerer
Docs: http://tinkerer.me/pages/documentation.html
Tinkerer is a very simple static blogging website generation tool written in Python
which extends Sphinx
and generates HTML-
from ReStructuredText
.
Static HTML pages generated with Tinkerer do not require a serverside application, and can be easily hosted with GitHub Pages or any other web hosting service.
- https://github.com/westurner/westurner.github.io/tree/source (
Makefile
,conf.py
)
Backup Tools
- https://en.wikipedia.org/wiki/Disk_cloning
- https://en.wikipedia.org/wiki/Disk_image#Virtualization
- https://en.wikipedia.org/wiki/List_of_archive_formats
- https://en.wikipedia.org/wiki/List_of_backup_software
- https://en.wikipedia.org/wiki/List_of_disk_cloning_software
- https://en.wikipedia.org/wiki/List_of_data_recovery_software
- https://en.wikipedia.org/wiki/Comparison_of_file_archivers
- https://en.wikipedia.org/wiki/Comparison_of_online_backup_services
- https://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software
Backup Ninja
Src: git git://labs.riseup.net/backupninja.git
Docs: https://labs.riseup.net/code/projects/backupninja/wiki
Docs: https://labs.riseup.net/code/projects/backupninja/wiki/Usage
Backup Ninja is an open source backup utility written in /etc/backup.d
- BackupNinja supports
rdiff-backup
,Duplicity
, andrsync
. - BackupNinja can create and burn CD/DVD images.
- BackupNinja can backup a number of relational databases (MySQL, PostgreSQL), maildirs, SVN repositories, Trac instances, and LDAP.
Bup
Src: git https://github.com/bup/bup
Docs: https://github.com/bup/bup/blob/master/README.md
Docs: https://bup.github.io/man.html
Docs: https://github.com/bup/bup/blob/master/DESIGN
Bup (backup) is a backup system based on git
packfiles and rolling checksums.
[Bup is a very] efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images).
- AFAIU, like
git
, Bup does not preserve file permissions, Access Control Lists, or extended attributes (though some archive formats and snapshot images do).
Clonezilla
DistroWatch: https://distrowatch.com/table.php?distribution=clonezilla
Homepage: https://clonezilla.org/
Src: git https://github.com/stevenshiau/clonezilla
Docs: https://clonezilla.org/clonezilla-live.php
Docs: https://clonezilla.org/clonezilla-SE/
Docs: https://clonezilla.org/related-links/
Clonezilla is an open source Linux
distribution which is bootable from a CD/DVD/USB (a LiveCD, LiveDVD, LiveUSB) or PXE which contains a number of tools for disk imaging, disk cloning, filesystem backup and recovery; and a server Linux
distribution for serving disk images to one or more computers over a LAN.
- Clonezilla contains
FSArchiver
,partclone
,partimage
, andrsync
. - Clonezilla can backup and restore very many (if not most) filesystems.
- Clonezilla supports MBR, GPT, and uEFI.
- Clonezilla can restore a networked multicast group (e.g. lab) of machines to a system image (saving TCP overhead when sharing the same multi-gigabyte / terabyte image to zero or more machines); and boot them with PXE and/or Wake-on-Lan.
bup
,debtorrent
- Clonezilla can backup to disk, ssh, samba, NFS, WebDAV
- drbl-winroll helps with restoring
windows
images SystemRescueCD
also containspartimage
.Cobbler
also supports PXE boot from images.
Duplicity
Homepage: http://duplicity.nongnu.org/
Docs: http://duplicity.nongnu.org/docs.html
Docs: http://duplicity.nongnu.org/duplicity.1.html
Docs: https://help.ubuntu.com/community/DuplicityBackupHowto
Docs: https://wiki.archlinux.org/index.php/Duplicity
Duplicity is an open source incremental file directory backup utility with GnuPG encryption, signatures, versions, and a number of actions for redundantly storing backups.
- Duplicity can push offsite backups to/over a number of protocols and services (e.g. SSH/SCP/SFTP, S3, Google Cloud Storage, Rackspace Cloudfiles (OpenStack Swift)).
- Duplicity stores data with tar archives and
rdiff
rdiff-backup
is similar toduplicity
.
FSArchiver
Homepage: https://www.fsarchiver.org/
Src: https://sourceforge.net/projects/fsarchiver/files/fsarchiver-src/
Download: https://sourceforge.net/projects/fsarchiver/files/fsarchiver-bin/
Docs: https://www.fsarchiver.org/QuickStart
Docs: https://www.fsarchiver.org/Live-backup
Docs: https://www.fsarchiver.org/Attributes#SELinux_.28Security_Enhanced_Linux.29
Docs: https://www.fsarchiver.org/Fsarchiver_vs_partimage
Docs: https://www.sysresccd.org/Sysresccd-manual-en_LVM_Making-consistent-backups-with-LVM
FSAchiver is an open source filesystem backup (disk cloning) utility which can preserve file permissions, labels, and extended attributes.
- FSArchiver can backup a filesysmet to a new or within an existing filesystem.
- FSArchiver has special support for LVM.
- FSArchiver supports password-based encryption.
partclone
Project: https://sourceforge.net/projects/partclone/
Download: https://partclone.org/download/
Src: git https://github.com/Thomas-Tsai/partclone
Docs: https://partclone.org/help/
Docs: https://partclone.org/usage/
Docs: https://github.com/Thomas-Tsai/partclone/wiki
partclone is an open source utility for making compressed backups of the used blocks of partitions with each specific filesystem driver.
partclone
is similar topartimage
.Clonezilla
includespartclone
.
partimage
Download: http://www.partimage.org/Download
Src: https://sourceforge.net/projects/partimage/files/stable/
Docs: http://www.partimage.org/Partimage-manual
Docs: http://www.partimage.org/Supported-Filesystems
Partimage is an open source utility for making complete sector-for-sector compressed backups of partitions over the network or to a local device.
Clonezilla
includespartimage
.- SystemRescueCD includes
partimage
andrsync
. - partimage does not support EXT4 or BTRFS; for EXT4 and BTRFS support, see
fsarchiver
.
rclone
Src: https://github.com/rclone/rclone
Homepage: https://rclone.org/
Docs: https://rclone.org/
Rclone is an open source utility for managing files on cloud storages like local disk, SFTP, WebDAV, Dropbox, and Google Drive.
- Rclone supports very many cloud storages
rsync
Homepage: https://rsync.samba.org/
Src: git git://git.samba.org/rsync.git
Download: https://rsync.samba.org/download.html
Docs: https://rsync.samba.org/examples.html
rsync is an open-source file backup utility which can be used to make incremental backups using file deltas over the network or the local system.
- rsync may appear to be stalled when it is actually calculating the full set of initial relative differences in order to minimize the amount of data transfer.
Note
rsync does not preserve file permissions by default.
To preserve file permissions with rsync:
man rsync
rsync -a # rsync -rlptgoD
rsync -r # recursive (traverse into directories)
rsync -l # copy symlinks as links
rsync -p # preserve file permissions
rsync -t # preserve modification times
rsync -g # preserve group
rsync -o # preserve owner (requires superuser)
rsync -D # rsync --devices --specials
rsync --devices # preserve device files (requires superuser)
rsync --specials # preserve special files
rsync -A # preserve file ACLs
rsync -X # preserve file extended attributes
rsync -aAX # rsync -a -A -X
rsync -v # verbose
rsync -P # rsync --partial --progress
rsync --partial # keep partially downloaded files
rsync --progress # show *per-file* progress and xfer speed
Note
rsync is picky about paths and trailing slashes.
# setUp
mkdir -p A/one B/one # TODO
echo 'A' > A/one; echo 'B' > B/one
# tests
rsync A B
rsync A B/ --> B/A
rsync A/ B
rsync A/ B/
rdiff
rdiff is the open source relative delta algorithm of rsync
.
rdiff-backup
is built onrdiff
.duplicity
is built onrdiff
rsnapshot
Download: https://rsnapshot.org/download.html
Download: https://rsnapshot.org/downloads/
Src: git https://github.com/rsnapshot/rsnapshot
Docs: https://rsnapshot.org/faq.html
Docs: https://wiki.archlinux.org/index.php/Rsnapshot
Docs: https://linux.die.net/man/1/rsnapshot
rsnapshot is an open source incremental file directory backup utility built with rsync
.
rdiff-backup
Download: https://download.savannah.gnu.org/releases/rdiff-backup/
Project: https://savannah.nongnu.org/projects/rdiff-backup
Src: svn svn://svn.savannah.nongnu.org/rdiff-backup/
Docs: https://nongnu.org/rdiff-backup/rdiff-backup.1.html
Docs: https://nongnu.org/rdiff-backup/docs.html
Docs: https://nongnu.org/rdiff-backup/examples.html
rdiff-backup is an open source incremental file directory backup utility.
- Like
rsync
, rdiff-backup transmits file deltas instead of entire files. - Unlike
rsync
, rdiff-backup manages reverting to previous revisions.
SystemRescueCD
Homepage: http://www.sysresccd.org/SystemRescueCd_Homepage
Download: http://www.sysresccd.org/Download
Project: https://sourceforge.net/projects/systemrescuecd/
DistroWatch: https://distrowatch.com/table.php?distribution=systemrescue
Docs: http://www.sysresccd.org/Online-Manual-EN
SystemRescueCD is a Linux
distribution which is bootable from a CD/DVD/USB (a LiveCD) which contains a number of helpful utilities for system maintenance.
- SystemRescueCD includes
partimage
andrsync
.
Standards
Docs: https://wrdrd.github.io/docs/consulting/knowledge-engineering#semantic-web-standards
Cascading Style Sheets
CSS
Docs: https://wrdrd.github.io/docs/consulting/knowledge-engineering#css
CSS (Cascading Style Sheets) define the presentational aspects of HTML-
and a number of mobile and desktop web framworks.
- CSS is designed to ensure separation of data and presentation. With javascript, the separation is then data, code, and presentation.
Filesystem Hierarchy Standard
Website: https://wiki.linuxfoundation.org/lsb/fhs
The Filesystem Hierarchy Standard (FHS) is a well-worn industry-supported system file naming structure.
Linux
distributions likeUbuntu
andFedora
implement a Filesystem Hierarchy.Likewise,
virtualenv
andVenv
implement a filesystem hierarchy:Docker
(and many LiveCDs) layer filesystem hierarchies with e.g. UnionFS, AUFS, and BTRFS filesystems.
HTTP
Docs: https://wrdrd.github.io/docs/consulting/knowledge-engineering#http
Docs: https://wrdrd.github.io/docs/consulting/knowledge-engineering#http2
HTTPS
Docs: https://wrdrd.github.io/docs/consulting/knowledge-engineering#https
HTML
Docs: https://wrdrd.github.io/docs/consulting/knowledge-engineering#html5
JSON
Homepage: https://json.org/
Spec: https://www.ecma-international.org/publications/standards/Ecma-404.htm
Docs: https://learnxinyminutes.com/docs/json/
JSON is an object representation in Javascript
syntax which is now supported by libraries written in many languages.
A list of objects with key
and value
attributes in JSON syntax:
[
{ "key": "language", "value": "Javascript" },
{ "key": "version", "value": 1 },
{ "key": "example", "value": true }
]
Machine-generated JSON is often not very readable, because it doesn't contain extra spaces or newlines. The Python
JSON library contains a utility for parsing and indenting ("prettifying") JSON from the commandline :
cat example.json | python -m json.tool
JSON5
Homepage: https://json5.org/
Src: https://github.com/json5/json5
Speci: https://spec.json5.org/
JSON5 is JSON extended with support for a number of additional features: comments, trailing commas, IEEE 754 +/- infinity and NaN, hexadecimal numbers, leading and trailing decimal points, single-quoted strings, multiline strings, and escaped characters.
- Regular JSON libraries do not support JSON5.
{
// comment
key: [0, +1, 2., .3, NaN, +inf, -inf, 0xF, 'thing1', "thing2"],
"str": "this is a \
multi-line string", // trailing comma
}
JSON-lines
JSON-lines (newline-delimited JSON) is an informal spec for line-based processing of JSON e.g. for streaming records and unix pipes.
{"key": "red", "value": 1}
{"key": "green", "value": 2}
JSONLD
JSON-LD
Homepage: https://json-ld.org
Docs: https://json-ld.org/playground/
Docs: https://wrdrd.github.io/docs/consulting/knowledge-engineering#json-ld
JSON-LD is a web standard for Linked Data in JSON <json->
.
An example from the JSON-LD Playground (https://goo.gl/xxZ410):
{
"@context": {
"gr": "http://purl.org/goodrelations/v1#",
"pto": "http://www.productontology.org/id/",
"foaf": "http://xmlns.com/foaf/0.1/",
"xsd": "http://www.w3.org/2001/XMLSchema#",
"foaf:page": {
"@type": "@id"
},
"gr:acceptedPaymentMethods": {
"@type": "@id"
},
"gr:hasBusinessFunction": {
"@type": "@id"
},
"gr:hasCurrencyValue": {
"@type": "xsd:float"
}
},
"@id": "http://example.org/cars/for-sale#tesla",
"@type": "gr:Offering",
"gr:name": "Used Tesla Roadster",
"gr:description": "Need to sell fast and furiously",
"gr:hasBusinessFunction": "gr:Sell",
"gr:acceptedPaymentMethods": "gr:Cash",
"gr:hasPriceSpecification": {
"gr:hasCurrencyValue": "85000",
"gr:hasCurrency": "USD"
},
"gr:includes": {
"@type": [
"gr:Individual",
"pto:Vehicle"
],
"gr:name": "Tesla Roadster",
"foaf:page": "http://www.teslamotors.com/roadster"
}
}
MessagePack
MessagePack (msgpack) is a data interchange format with implementations in many languages.
MsgPack
is a Distributed Computing Protocol https://wrdrd.github.io/docs/consulting/knowledge-engineering#distributed-computing-protocolsSalt
serializes messages withMsgPack
by default.
Text Editors
Wikipedia: https://en.wikipedia.org/wiki/Source_code_editor
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Free_text_editors
- A text editor may be a source code editor or have some
IDE
features like syntax highlighting or syntax checking forProgramming Languages
. - Most
IDEs
are source code text editors. - https://en.wikipedia.org/wiki/Comparison_of_text_editors
- https://en.wikipedia.org/wiki/List_of_text_editors
Gedit
Homepage: https://wiki.gnome.org/Apps/Gedit
Src: git https://gitlab.gnome.org/GNOME/gedit/
Gedit is an open source text editor written in C
and Python
(GTK
, GtkSourceView, and Gnome
) that's available for Linux
, OSX
, and Windows
).
- Gedit supports tabbed editing.
- Gedit plugins are written in
Python
. - Gedit is the default
Gnome
text editor; where it's called "Text Editor".
Notepad++
Homepage: https://notepad-plus-plus.org/
Src: https://github.com/notepad-plus-plus/notepad-plus-plus
Docs: https://notepad-plus-plus.org/online-help/
ChocolateyPackage: https://chocolatey.org/packages/notepadplusplus
Notepad++ is an open source text editor written in C++
for Windows
which has tabbed editing.
- Notepad++ supports tabbed editing.
- Notepad++ plugins are written in
C
orC++
: - npppythonscript is a plugin that enables scripting Notepad++ with
Python
- Notepad++ was the most used editor according to a 2015 Stack OVerflow survey.
IDEs
Integrated Development Environments
WikipediaCategory: https://en.wikipedia.org/wiki/Category:Integrated_development_environments
An IDE (Integrated Development Environment) is a software tool for developing software.
- Most IDEs are source code
Text Editors
. - Some IDEs are visual development tools for various types of not code trees and graphs.
- IDEs have a concept of a project, which may be defined in a config file in the current working directory or otherwise selected through the GUI.
- An IDE has some sort of language server that understands the source code at a deeper level than syntax in order to do cool things like code completion and code refactorings like renaming a method in every file in the project.
- https://en.wikipedia.org/wiki/Comparison_of_integrated_development_environments
Emacs
GNU Emacs
Homepage: https://www.gnu.org/software/emacs/
Src: git https://git.savannah.gnu.org/cgit/emacs.git
Docs: https://www.gnu.org/software/emacs/documentation.html
GNU Emacs is an open source text editor written in Emacs Lisp
and C
that's available for Linux
, OSX
, and Windows
.
- Emacs pinky is allegedly a result of the default emacs
Control
key keybindings https://en.wikipedia.org/wiki/Emacs#Emacs_pinky Spacemacs
uses theSpace
key instead of theControl
key.
Spacemacs
Homepage: https://wwww.spacemacs.org/
Src: https://github.com/syl20bnr/spacemacs
Docs: https://www.spacemacs.org/doc/DOCUMENTATION
Docs: https://wwww.spacemacs.org/doc/QUICK_START
Print help:
SPC h SPC
org-mode
Homepage: https://orgmode.org/
Src: git https://code.orgmode.org/bzg/org-mode
Docs: https://orgmode.org/#docs
Docs: https://orgmode.org/org.html
Docs: https://orgmode.org/org.html#Introduction
Org-mode is an open source document editing mode originally written in Emacs Lisp
for Emacs
that's now available in some form for a number of editors including Vim
.
- Org-mode makes it really easy to work with outlines in plain text documents.
- The org-mode wikipedia page lists a number of org-mode implementations for other editors.
org-babel
Homepage: https://orgmode.org/worg/org-contrib/babel/
Src: git https://code.orgmode.org/bzg/org-mode
Docs: https://orgmode.org/manual/Working-with-source-code.html
- Babel makes it possible to execute source code in org-mode
- Babel is also the name of an
ECMAScript
compiler Jupyter Notebook
withJupytext
and/or emacs and vim plugins for working with Jupyter are similar to Babel org-mode.
VSCode
Homepage: https://code.visualstudio.com/
Src: https://github.com/Microsoft/vscode
Download: https://code.visualstudio.com/Download
ChocolateyPkg: https://chocolatey.org/packages/vscode
Docs: https://code.visualstudio.com/docs
Docs: https://code.visualstudio.com/docs/getstarted/tips-and-tricks
Docs: https://code.visualstudio.com/docs/getstarted/keybindings
VSCode (Visual Studio Code) is an open source programmer's text editor written in TypeScript
, Javascript
, and CSS
that's available for Windows
, Mac
, and Linux
.
- VSCode extensions are written in
Javascript
. - VSCode has collaborative editing features with multiple cursors.
- VSCode and MS Visual Studio are different projects.
- VSCode supports many of the Visual Studio keyboard shortcuts.
- There is an official
Vim
extension forVSCode
. - In VSCode,
Ctrl+Space
opens the context-sensitive Intellisense Code Completion - In VScode,
Ctrl-p
opens the quick open dialogue - IN VScode,
Ctrl-Shift-p
opens the command palette (which lists "all available commands based on your current context")
You can install VSCode by downloading from the Download page or with Chocolatey
:
choco install vscode
Vim
Homepage: https://www.vim.org/
Docs: https://www.vim.org/docs.php
Docs: https://learnxinyminutes.com/docs/vim/
Src: git https://github.com/vim/vim
ViM (VI-iMproved) is an open source text editor written in C
that's available on very many platforms.
- Vim help can be accessed with
:help
and:help help
(Press<esc>
, Type:help help
, Press Enter) - Vi is almost always installed on Linux and BSD boxes.
- Vi is often included with
Busybox
. - Vi and Vim are installed with
OSX
. - Vi and Vim are installed by default with many
Linux Distributions
- Vim runs in a terminal, over SSH, and with a GUI window manager (Gvim, Macvim)
- Vim configuration is written in the vim language.
- Vim reads a few vimrc configuration files in sequence (
:help vimrc
) - GVim is Vim for
Gnome
window manager - GVim reads a few vimrc configuration files in sequence (
:help gvimrc
) - MacVim is Vim for
OSX
- One way to write changes and exit vim:
:wq!
(Press<esc>
, Type:wq!
, Press Enter) - There are many plugins for vim.
- NERDTree is an example of a vim plugin: https://github.com/scrooloose/nerdtree (
:help nerdtree
) SpaceVim
andwesturner/dotvim
include the NERDtree plugin- Vim keyboard shortcuts are calling mappings.
- Vim mappings are defined in a vimrc file.
- Examples of vim mappings:
\e
opens NERDTree,\E
opens NERDTree to the current file - Vim mappings can be defined for different vim modes:
:map \e
(command mode),:imap \e
(insert mode) (:help modes
) - Press
i
ora
while in command mode to enter insert or append mode (:help vim-modes
) - Press
<Esc>
to return to command mode
Browser extensions
with vim-style keyboard shortcuts:
Vimium
(Chrome)Vimperator
(Firefox)Wasavi
(Chrome, Opera, Firefox)
A number of web apps support vim-style keyboard shortcuts like j
and k
for up and down:
- GMail (
?
for help) - Facebook (
?
for help) - Twitter (
?
for help)
westurner/dotvim
Src: https://github.com/westurner/dotvim
Src: https://github.com/westurner/dotvim/blob/master/Makefile
Src: https://github.com/westurner/dotvim/blob/master/vimrc
Src: https://github.com/westurner/dotvim/blob/master/vimrc.full.bundles.vimrc
Src: https://github.com/westurner/dotvim/blob/master/vimrc.tinyvim.bundles.vimrc
westurner/dotvim is a set of plugins and configuration defaults for Vim
.
SpaceVim
Src: https://github.com/SpaceVim/SpaceVim
Docs: https://spacevim.org/documentation/
Docs: https://spacevim.org/documentation/#general-key-bindings
SpaceVim is a set of plugins, configuration defaults, and keybindings for Vim
.
Spacemacs
is like SpaceVim forEmacs
WRD R&D Documentation
Docs: https://wrdrd.github.io/docs/tools/
Src: git https://github.com/wrdrd/docs