Merge pull request #2479 from sanssecours/📚

Documentation: Add Basic Storage Plugin Tutorial

doc/markdownlinkconverter/elektraSpecialCharacters.sty

@@ -17,3 +17,5 @@
 \newunicodechar{🍏}{green apple}
 \newunicodechar{🍎}{red apple}
 \newunicodechar{🐧}{penguin}
+\newunicodechar{🐣}{hatching chick}
+\newunicodechar{🐓}{cockerel}

doc/news/2016-09-17_0.8.18.md

@@ -33,7 +33,7 @@ for encryption of decryption of individual keys is present.
 Furthermore, a new [botan](https://botan.randombit.net) backend
 was implemented.
 
-[See here](https://git.libelektra.org/tree/master/src/plugins/crypto)
+[See here](https://master.libelektra.org/src/plugins/crypto)
 
 Thanks to Peter Nirschl.
 
@@ -51,7 +51,7 @@ Elektra provides.
 
 For easy setup, we implemented the script `configure-firefox`.
 
-[See here](https://git.libelektra.org/tree/master/src/bindings/intercept)
+[See here](https://master.libelektra.org/src/bindings/intercept)
 
 Thanks to Thomas Waser.
 
@@ -61,10 +61,10 @@ Resolvers in Elektra are the code that are responsible to determine where
 content should be read from and stored to. They are independent of the
 actual configuration file syntax.
 
-The [gitresolver](https://git.libelektra.org/tree/master/src/plugins/gitresolver)
+The [gitresolver](https://master.libelektra.org/src/plugins/gitresolver)
 allows you to get/store config data in git.
 
-The [blockresolver](https://git.libelektra.org/tree/master/src/plugins/blockresolver)
+The [blockresolver](https://master.libelektra.org/src/plugins/blockresolver)
 allows Elektra to take control of parts of the configuration
 file. This is useful for config files such as vim or zsh, which contain
 program code. The plugin allows you to split config files with special markers
@@ -136,7 +136,7 @@ specify which relation should be checked.
 - github descriptions+workflow (displayed by github when creating
   PRs and issues)
 - new trigger phases for github, see
-  [doc/GIT](https://git.libelektra.org/tree/master/doc/GIT.md)
+  [doc/GIT](https://master.libelektra.org/doc/GIT.md)
   thanks to Mihael Pranjić
 - valgrind suppressions are great again, thanks to Peter Nirschl
 - Plugins get a new namespace `internal` which can be used for meta-data
@@ -181,9 +181,9 @@ specify which relation should be checked.
 - first data structures for order preserving minimal perfect hash map,
   thanks to Kurt Micheli
 - added a new passwd plugin, thanks to Thomas Waser
-- [boolean](https://git.libelektra.org/tree/master/src/plugins/boolean)
+- [boolean](https://master.libelektra.org/src/plugins/boolean)
   plugin to normalize boolean values, thanks to Thomas Waser
-- [desktop](https://git.libelektra.org/tree/master/src/plugins/desktop)
+- [desktop](https://master.libelektra.org/src/plugins/desktop)
   plugin to detect which desktop currently is running (supports kde,
   gnome, tde, unity or any other XDG conformant desktop)
 - `doc/paper` contains some info for [joss](https://github.com/openjournals/joss)

doc/news/_preparation_next_release.md

@@ -185,6 +185,7 @@ you up to date with the multi-language support provided by Elektra.
 
 ## Documentation
 
+- We added a (very) basic tutorial that tells you [how to write a (well behaved) storage plugin](../tutorials/storage-plugins.md). _(René Schwaiger)_
 - The documentation now uses [fenced code blocks](https://help.github.com/en/articles/creating-and-highlighting-code-blocks#syntax-highlighting) to improved the syntax highlighting of code snippets. _(René Schwaiger)_
 - The [Markdown Link Converter](https://master.libelektra.org/doc/markdownlinkconverter) now uses the style
 

doc/tutorials/compilation-variants.md

@@ -120,4 +120,4 @@ compiled included the `#ifdef` the other (base variant called
 `myplugin`) without.
 
 Currently compilation variants are used in
-[the resolver plugin](https://libelektra.org/tree/master/src/plugins/resolver/resolver.c).
+[the resolver plugin](https://master.libelektra.org/src/plugins/resolver/resolver.c).

doc/tutorials/plugins.md

@@ -4,7 +4,7 @@ This file serves as a tutorial on how to write a storage plugin (which includes
 
 ## Types of Plugins
 
-- Storage plugins are used by Elektra in order to store data in the Elektra Key Database
+- [Storage plugins](storage-plugins.md) are used by Elektra in order to store data in the Elektra Key Database
   in an intelligent way. They act as a liaison between configuration files and the Key Database. Storage plugins are largely responsible for
   the functionality of Elektra and they allow many of its advanced features to work.
   These plugins act as sources and destinations of configuration settings.
@@ -32,7 +32,7 @@ All plugins use the same basic interface. This interface consists of five basic
 `elektraLineOpen()`, `elektraLineGet()`, `elektraLineSet()`, `elektraLineError()`, and `elektraLineClose()`.
 Additionally, there is one more function called
 [ELEKTRA_PLUGIN_EXPORT](https://doc.libelektra.org/api/current/html/group__plugin.html#ga8dd092048e972a3f0c9c9f54eb41576e),
-where once again `Plugin` should be replaced with the name of the plugin, this time in uppercase. So for my line plugin this function would be
+where once again `Plugin` should be replaced with the name of the plugin, this time in uppercase. So for the line plugin this function would be
 `ELEKTRA_PLUGIN_EXPORT(line)`.
 The developer may also define `elektraPluginCheckConf()` if configuration validation at mount-time is desired.
 
@@ -48,15 +48,15 @@ that allow the plugin to work:
 - `elektraPluginError()` is designed to allow proper rollback of operations if needed and is called if any plugin fails during the set operation.
   This is not needed for storage plugins as the resolver already takes care to unlink the configuration files in such situations.
 - `elektraPluginClose()` is used to free resources that might be required for the plugin.
-- `ELEKTRA_PLUGIN_EXPORT(Plugin)` simply lets Elektra know that the plugin exists and what the name of the above functions are.
+- `ELEKTRA_PLUGIN_EXPORT` simply lets Elektra know that the plugin exists and what the name of the above functions are.
 
 Most simply put: most plugins consist of five major functions, `elektraPluginOpen()`, `elektraPluginClose()`, `elektraPluginGet()`, `elektraPluginSet()`,
-and `ELEKTRA_EXPORT_PLUGIN(Plugin)`.
+and `ELEKTRA_EXPORT_PLUGIN`.
 
 Because remembering all these functions can be cumbersome, we provide a skeleton plugin in order to easily create a new plugin.
 The skeleton plugin is called [`template`](/src/plugins/template/) and a new plugin can be created by calling the
 [copy-template script](/scripts/copy-template) .
-For example for my plugin I called `scripts/copy-template line`. Afterwards two
+For example, the author of the [line plugin](/src/plugins/line/) used the command `scripts/copy-template line` to create the initial version of the plugin. Afterwards two
 important things are left to be done:
 
 - remove all functions (and their exports) from the plugin that are not needed. For example not every plugin actually makes use of the `elektraPluginOpen()` function.
@@ -97,7 +97,7 @@ generate_readme(pluginname)
 ```
 
 It will generate a `readme_plugginname.c` (in the build-directory) out of the
-README.md of the plugin’s source directory.
+`README.md` of the plugin’s source directory.
 
 But prefer to use
 
@@ -136,7 +136,7 @@ All these clauses need to be present for every plugin.
 
 The information of clauses are limited to a single line, starting with
 `-` (so that the file renders nicely in Markdown), followed by the clause
-itself separated with `=`.
+itself separated by `=`.
 Only for the description an unlimited amount of lines can be
 used (until the end of the file).
 
@@ -169,7 +169,7 @@ if (!strcmp (keyName(parentKey), "system/elektra/modules/plugin"))
 
 The `elektraPluginContract()` is a method implemented by the plugin developer
 containing the parts of the contract not specified in `README.md`.
-An example of this function (taken from the `yajl` plugin):
+An example of this function (taken from the [`yajl`](/src/plugins/yajl/) plugin):
 
 ```c
 static inline KeySet *elektraYajlContract()
@@ -349,7 +349,7 @@ for (/* each key */)
 // close the file
 ```
 
-The full-blown code can be found at [line plugin](https://libelektra.org/tree/master/src/plugins/line/line.c).
+The full-blown code can be found at [line plugin](https://master.libelektra.org/src/plugins/line/line.c).
 
 As you can see, all `elektraLineSet` does is open a file, take each `Key` from the `KeySet` (remember they are named `#1`, `#2` ... `#_22`) in order,
 and write each key as its own line in the file. Since we don't care about the name of the `Key` in this case (other than for order), we just write
@@ -420,7 +420,7 @@ The last function, one that is always needed in a plugin, is `ELEKTRA_PLUGIN_EXP
 the plugin exists and which methods it implements. The code from the line plugin is a good example and pretty self-explanatory:
 
 ```c
-Plugin *ELEKTRA_PLUGIN_EXPORT(line)
+Plugin *ELEKTRA_PLUGIN_EXPORT
 {
 	return elektraPluginExport("line",
 	ELEKTRA_PLUGIN_GET, &elektraLineGet,

doc/tutorials/storage-plugins.md

@@ -0,0 +1,106 @@
+# How-To: Write a (Well Behaved) Storage Plugin
+
+The [plugin tutorial](plugins.md) already covers some of the most interesting parts on how to write a (storage) plugin. This text will tell you a little bit more about how a storage plugin should act. While it is usually relatively easy to create a plugin that stores basic key-value pairs, adding advanced features such as support for
+
+- [arrays](arrays.md), and
+- [metadata](../dev/metadata.md),
+
+takes more work. Before you continue with this text, please make sure that you read all of the linked documents above.
+
+## Don’t Add Additional Keys
+
+One common problem of storage plugins is, that they store too many keys. For example, if the user adds the keys
+
+- `user/tests/storage/root` and
+- `user/tests/storage/root/level1/level2/level3`,
+
+then your plugin should only store those two keys. **Do not** add the keys
+
+- `user/tests/storage/root/level1`, or
+- `user/tests/storage/root/level1/level2`
+
+to the key set. One plugin that handles this situation properly is [YAML CPP](/src/plugins/yamlcpp/), as the following [Markdown Shell Recorder test](https://master.libelektra.org/tests/shell/shell_recorder/tutorial_wrapper) shows:
+
+```sh
+# Mount plugin
+sudo kdb mount config.yaml user/tests/storage yamlcpp
+
+# Add key-value pairs
+kdb set user/tests/storage/root 🐓
+kdb set user/tests/storage/root/level1/level2/level3 🐣
+
+# Make sure that YAML CPP did not store any additional keys
+kdb ls user/tests/storage/root
+#> user/tests/storage/root
+#> user/tests/storage/root/level1/level2/level3
+
+# Undo modifications to the key database
+sudo kdb umount user/tests/storage
+```
+
+. For more information on why we allow “holes” in the hierarchy, please take a look [here](../decisions/holes.md).
+
+## Support Values Inside Non-Leaf Keys
+
+Sometimes the most “natural” mapping of key-value pairs to a file format might cause a storage plugin to not be able to store values in so called directory (non-leaf) keys.
+
+For example, in a key set that contains the keys:
+
+```
+user/directory
+user/directory/leaf1
+user/directory/leaf2
+user/leaf3
+```
+
+, all keys at the bottom of the hierarchy:
+
+```
+        user
+      /      \
+  directory  leaf3
+   /     \
+leaf1   leaf2
+```
+
+, such as
+
+- `user/directory/leaf1`
+- `user/directory/leaf2`
+- `user/leaf3`
+
+are called leaf keys, while `user/directory` is a directory key. Plugins such as [YAJL](/src/plugins/yajl/) or [YAML CPP](/src/plugins/yamlcpp/) will not be able to store data in the key with the name `user/directory` directly. To work around this issue these plugin use the [Directory Value plugin](/src/plugins/directoryvalue/). In the ReadMe of the [Directory Value plugin](https://www.libelektra.org/plugins/directoryvalue) and [YAML CPP](https://www.libelektra.org/plugins/yamlcpp) you will find more information about this issue, and how to handle it.
+
+The following Markdown Shell Recorder test shows **the proper behavior**:
+
+```sh
+# Mount plugin
+sudo kdb mount config.yaml user/tests/storage yamlcpp
+
+# Add key-value pair (leaf key)
+kdb set user/tests/storage/root 🐓
+# Since we add a key below `user/tests/storage/root`, the key
+# `user/tests/storage/root` turns from a leaf key to a directory key.
+kdb set user/tests/storage/root/level1/level2/level3 🐣
+
+# Make sure that the directory key still stores the correct value
+kdb get user/tests/storage/root
+#> 🐓
+
+# Check the value of the leaf key
+kdb get user/tests/storage/root/level1/level2/level3
+#> 🐣
+
+# Undo modifications to the key database
+sudo kdb umount user/tests/storage
+```
+
+. To make sure that your storage plugin works correctly, please just replace `yamlcpp` with the name of your plugin and verify that the test above still works.
+
+<!--
+TODO: Describe difference between keys that store null values (binary data) and empty values (string data)
+TODO: Describe how to store arrays properly (`array` metadata): Just add a link to `arrays.md`
+TODO: Add information on how plugins should store comment (meta)data
+TODO: Document that a plugin should keep the ordering of key-value pairs of a document intact, when writing data back to the configuration file
+TODO: Add section about relative keys (See also: https://issues.libelektra.org/51)
+-->

src/libs/typesystem/specelektra/specelektra.cabal.in

@@ -6,7 +6,7 @@ description:         A type system for libelektra based on regular expressions.
                      the regexes are compatible with each other, e.g. if a Key a falls back to a
                      Key b, this is only possible if the regex representing Key b is contained or
                      equal to the regex representing the content of Key a.
-homepage:            https://git.libelektra.org/tree/master/src/plugins/typechecker
+homepage:            https://master.libelektra.org/src/plugins/typechecker
 license:             BSD3
 author:              Armin Wurzinger
 maintainer:          e1528532@libelektra.org

src/tools/rest-backend/README.md

@@ -135,7 +135,7 @@ An extensive tutorial describing the installation and configuration can be found
 
 ### Compiling
 
-Compile Elektra as normal as per the [COMPILE document](https://libelektra.org/tree/master/doc/COMPILE.md),
+Compile Elektra as normal as per the [COMPILE document](https://master.libelektra.org/doc/COMPILE.md),
 but make sure to include the `rest-backend` tool using the `-DTOOLS` flag.
 
 For instance:
@@ -144,7 +144,7 @@ For instance:
 ### Installing
 
 You can now install Elektra as you normally would or as described
-in the [install documentation](https://libelektra.org/tree/master/doc/INSTALL.md).
+in the [install documentation](https://master.libelektra.org/doc/INSTALL.md).
 
 ## Implementation notes and hints for Front-Ends
 
@@ -163,7 +163,7 @@ This limitation comes from the usage of regex patterns instead of atomic compari
 In terms of usability this is sufficient, but not the best possible.
 Therefore it would be advisable to implement live-validation for frontends with more granularity.
 Information about allowed input formats can be found in the
-[API description](https://libelektra.org/tree/master/doc/rest_api/snippet_sharing/api-description.apib).
+[API description](https://master.libelektra.org/doc/api_blueprints/snippet-sharing.apib).
 
 ## Benchmarks
 

src/tools/rest-backend/config-specification.ini.in

@@ -4,7 +4,7 @@ mountpoint = rest-backend.ini
 [@config_default_profile@/backend/api/description/raw]
 check/type = string
 description = A link to the blueprint describing the API.
-default = https://git.libelektra.org/tree/master/doc/api_blueprints/snippet-sharing.apib
+default = https://master.libelektra.org/doc/api_blueprints/snippet-sharing.apib
 
 [@config_default_profile@/backend/api/description/html]
 check/type = string
@@ -176,7 +176,7 @@ default = 65536
 
 [@config_default_profile@/cppcms/security/file_in_memory_limit]
 check/type = long
-description = When files are uploaded for efficiency, small files are generally stored in memory and big ones are saved in files. This is the limit on the file size to be stored in memory in bytes. 
+description = When files are uploaded for efficiency, small files are generally stored in memory and big ones are saved in files. This is the limit on the file size to be stored in memory in bytes.
 default = 128
 
 [@config_default_profile@/cppcms/security/uploads_path]

src/tools/web/README.md

@@ -3,7 +3,7 @@
 _an API and web user interface to remotely manage Elektra instances_
 
 The configuration view of elektra-web is similar to the tree view of the
-[qt-gui](https://git.libelektra.org/tree/master/src/tools/qt-gui), but with
+[qt-gui](https://master.libelektra.org/src/tools/qt-gui), but with
 dynamic fields rendered via key metadata.
 
 ## Dependencies

tests/shell/shell_recorder/tutorial_wrapper/CMakeLists.txt

@@ -24,6 +24,7 @@ endif (CMAKE_COMPILER_IS_GNUCXX AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 4.9)
 
 add_msr_test (tutorial_arrays "${CMAKE_SOURCE_DIR}/doc/tutorials/arrays.md")
 add_msr_test (tutorial_cascading "${CMAKE_SOURCE_DIR}/doc/tutorials/cascading.md")
+add_msr_test (tutorial_storage_plugins "${CMAKE_SOURCE_DIR}/doc/tutorials/storage-plugins.md" REQUIRED_PLUGINS yamlcpp)
 
 if (ENABLE_ASAN)
 	message (STATUS "Excluding Markdown Shell Recorder test for `validation`, as it leaks memory and fails with ASAN enabled")