Skip to content
This repository has been archived by the owner on Oct 15, 2024. It is now read-only.

Hexnumber #1903

Merged
merged 18 commits into from
May 19, 2018
Merged

Hexnumber #1903

merged 18 commits into from
May 19, 2018

Conversation

kodebach
Copy link
Member

@kodebach kodebach commented Apr 12, 2018

Purpose

Implementation of hexnumber plugin

Checklist

Check relevant points but please do not remove entries.
For docu fixes, spell checking, and similar nothing
needs to be checked.

  • commit messages are fine ("module: short statement" syntax and refer to issues)
  • I added unit tests
  • I ran all tests locally and everything went fine
  • affected documentation is fixed
  • I added code comments, logging, and assertions (see doc/CODING.md)
  • meta data is updated (e.g. README.md of plugins)
  • release notes are updated (doc/news/_preparation_next_release.md)

@markus2330 please review my pull request

Copy link
Contributor

@markus2330 markus2330 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clean and nice code but some design choices are to be discussed. Please prefer to first discuss the design of meta data before implementing a full plugin.

Please also make sure to write a line about your changes in the release notes.

@@ -573,7 +573,7 @@ function (generate_readme p)
STRING(REGEX REPLACE "\n" "\\\\n\"\n\"" contents "${contents}")
STRING(REGEX REPLACE "- infos = ([a-zA-Z0-9 ]*)\\\\n\"" "keyNew(\"system/elektra/modules/${p}/infos\",\nKEY_VALUE, \"\\1\", KEY_END)," contents "${contents}")
STRING(REGEX REPLACE "\"- +infos/licence *= *([a-zA-Z0-9 ]*)\\\\n\"" "keyNew(\"system/elektra/modules/${p}/infos/licence\",\nKEY_VALUE, \"\\1\", KEY_END)," contents "${contents}")
STRING(REGEX REPLACE "\"- +infos/author *= *([.@<>a-zéA-Z0-9 %_-]*)\\\\n\"" "keyNew(\"system/elektra/modules/${p}/infos/author\",\nKEY_VALUE, \"\\1\", KEY_END)," contents "${contents}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As said, everything that is invalid within C-strings, must be invalid, too. E.g. ".

## Introduction

This plugin can be used to read configuration files that use hexadecimal values. All hexadecimal values (strings starting with 0x) will be
converted into decimal when the Elektra reads values from the mounted file. When Elektra writes back to the file all values originally
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

long sentence

- To mount a simple backend that uses hexadecimal numbers, you can use:

```sh
sudo kdb mount test.ecf /examples/hexnumber/test hexnumber
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please give an example.

```sh
sudo kdb umount /examples/hexnumber/test
```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are there any limitations?


This plugin can be used to read configuration files that use hexadecimal values. All hexadecimal values (strings starting with 0x) will be
converted into decimal when the Elektra reads values from the mounted file. When Elektra writes back to the file all values originally
stored as hexadecimal will be converted back and stored as before.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if a string starts with 0x and is not a number? (0xX) Shouldn't we use type and check/type to be sure it is a number?

Are lower and upper characters A-F allowed? Is 0X allowed?

/**
* Establish the plugin contract and convert all hexadecimal values in the KeySet to decimal.
*
* @note The plugin will attempt to convert ALL values starting with 0x from hexadecimal into decimal.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes here will not be read by the user of the plugin.

The feature seems to dangerous, see other discussions and #1398.

int status = ELEKTRA_PLUGIN_STATUS_NO_UPDATE;
while ((cur = ksNext (returned)) != NULL)
{
if (!keyIsString (cur) || !isHexString (cur))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please invert the expression, then you get rid of continue.

int status = ELEKTRA_PLUGIN_STATUS_NO_UPDATE;
while ((cur = ksNext (returned)) != NULL)
{
if (!keyIsString (cur) || !hasHexType (cur))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, please invert logic.


#include <kdbplugin.h>

#define ELEKTRA_HEXNUMBER_META_TYPE "hexnumber"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have to separate meta data for internal purposes and for the user to be specified.

  1. To remember what you have transformed to transform it back please use meta keys in internal/<plugin>
  2. For what the user needs to specify, we should discuss this in units-of-measurements plugin #1398 first.

For non-internal meta-data please add the specification in doc/METADATA.ini and in the plugin's contract.

init (argc, argv);

test_basics ();
test_default ();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check more for errors and the validation parts of this plugin.

@@ -0,0 +1,31 @@
- infos = Information about the template plugin is in keys below
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please replace ”template“ with the name of the plugin (“hexnumber“).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

The ../../scripts/copy-template approach avoids such problems.

@markus2330
Copy link
Contributor

Is this ready for review? (remove "work in progress" if it is) Please rebase after the release.

@kodebach
Copy link
Member Author

@markus2330 everything you noted in your original review should be fixed now, please review again

I am not removing the "work in progress" tag, because the testshell_markdown_hexnumber test still fails, but I am not sure why, because manually running the code in the README.md file works fine.
What exactly does the test do other than executing the code?

@markus2330
Copy link
Contributor

The "work in progress" basically only means "I am still working on it, please do not review". If build jobs do not work is a separate problem (except you are still trying to fix the build jobs)

The build server might have less packages and environment variables. Furthermore, it is guaranteed to make a clean build with only the files as pushed in the repo.

The first step to reproduce is to rebuild everything from scratch locally.

@sanssecours
Copy link
Member

sanssecours commented May 14, 2018

What exactly does the test do other than executing the code?

Basically the Shell Recorder executes each line via sh -c. To learn more check out markdown_shell_recorder.sh and shell_recorder.sh.

I executed some of the commands from the ReadMe in my shell. While the following works:

kdb mount test.ecf user/examples/hexnumber/test hexnumber
kdb set user/examples/hexnumber/test/hex 0x1F
kdb get user/examples/hexnumber/test/hex
#> 0x1F

, the commands:

kdb mount test.ecf user/examples/hexnumber/test hexnumber
kdb set user/examples/hexnumber/test/hex 0x1F

kdb setmeta user/examples/hexnumber/test/hex type int
kdb set user/examples/hexnumber/test/dec 26 # Without this command everything works fine
kdb get user/examples/hexnumber/test/hex # This command should print the value “31”
#> 1

produce unexpected output.

By the way: We decided to save all test data below /tests. Please replace user/examples with user/tests accordingly.

@markus2330
Copy link
Contributor

Thank you for mentioning /tests.

Btw. why are you linking to the shell scripts and not the accompanied README.md? How can we improve the README.md?

@sanssecours
Copy link
Member

Btw. why are you linking to the shell scripts and not the accompanied README.md?

Because the code tells you exactly how the tests work (including all the warts).

How can we improve the README.md?

I assume the best way would be to let someone without any knowledge about the (Markdown) Shell Recorder write a Markdown Shell Recorder test. Afterwards this person could then update and extend the ReadMe, adding information that might not be obvious. Another person with extended knowledge of the MSR (either you or Thomas 😊) could then write a review of the extended ReadMe correcting wrong, unclear and outdated information.

@kodebach
Copy link
Member Author

kodebach commented May 14, 2018

I found the bug, everything should be working now.

The "work in progress" basically only means "I am still working on it, please do not review"

@markus2330 I also do not have the access rights to remove the label.

How can we improve the README.md?

@sanssecours explanation matches what I understood from the README, so I don't think we need to change anything.

@markus2330
Copy link
Contributor

I also do not have the access rights to remove the label.

You should now have the rights! You were added to ElektraDevelopers

Copy link
Contributor

@markus2330 markus2330 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, it is nearly finished, most comments are about docu.

doc/METADATA.ini Outdated
[unit/base]
type = enum dec bin hex any
default = dec
description = allows values to be written in other bases.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-v please, what exactly does this metadata do.

Please also say that it is used by the hexnumber plugin

@@ -59,7 +59,8 @@ We added even more functionality, which could not make it to the highlights:

## New Plugins

- <<TODO>>
- The plugin [hexnumber](https://www.libelektra.org/plugins/hexnumber) has been added. It can be used
to hexadecimal values into decimal when read, and back to hexadecimal when written.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some verb missing. What can it be used for?


## Introduction

This plugin can be used to read configuration files that use hexadecimal values. All "hex-values" (see below) will be
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid modal verbs "can be" -> "is"

### What are "hex-values"?
There are multiple ways you can signal to the hexnumber plugin, that a value should be converted:

1. If a Key has the metadata `/unit/base` set to `hex` it will always be interpreted as a hex-value. The plugin will also produce an error,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unit/base should be without slash in the beginning


1. If a Key has the metadata `/unit/base` set to `hex` it will always be interpreted as a hex-value. The plugin will also produce an error,
if the value contained in such a Key does not start with `0x` (or `0X`).
2. If a Key has the metadata `type` set to an integer-type and its value starts with `0x` (or `0X`) it will be interpreted as a hex-value.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a Key has the metadata type and unit/base is not present...


# Set up examples
kdb set user/tests/hexnumber/hex 0x1F
kdb setmeta user/tests/hexnumber/hex type int
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

int is not a CORBA type

kdb set user/tests/hexnumber/hex2 0xF
kdb setmeta user/tests/hexnumber/hex2 unit/base hex

# Example 1: read hex value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is easier to read if the get is directly after the respective set.

++ptr;
}
*array = elektraCalloc ((count + 1) * sizeof (char *));
char * localString = strdup (values);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use elektraStrDup (for everything that allocates memory, there is an elektra* method)

const char * type = keyString (typeMeta);
for (int i = 0; types[i] != NULL; ++i)
{
if (strcmp (types[i], type) == 0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use elektraStrCmp

*/
static bool isHexString (const Key * key)
{
return strncasecmp (keyString (key), "0x", 2) == 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use elektraStrCaseCmp

@kodebach
Copy link
Member Author

@markus2330 everything should be fixed now, please review

@@ -251,76 +255,31 @@ static bool hasHexType (const Key * key)
return typeMeta && elektraStrCmp (keyString (typeMeta), "1") == 0;
}

/**
* copied from boolean/boolean.c
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for getting rid of this! ❤️

Copy link
Member

@sanssecours sanssecours left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I especially like the detailed documentation. The failing Travis build jobs should be fixed after

.

ksDel (elements);

return size;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not remove the newline at the end of the file:

src/libs/ease/array.c:300:2: error: no newline at end of file [-Werror,-Wnewline-eof]
}
 ^
1 error generated.

.

## Introduction

This plugin is used to read configuration files that use hexadecimal values. All "hex-values" (see below) will be
converted into decimal when the Elektra reads values from the mounted file. When Elektra writes back to the file the converted values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: the ElektraElektra

return status;
}

int elektraHexnumberClose (Plugin * handle ELEKTRA_UNUSED, Key * parentKey ELEKTRA_UNUSED)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove ELEKTRA_UNUSED after the argument handle.

Copy link
Contributor

@markus2330 markus2330 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor things to be changed together with @sanssecours requests.

@sanssecours Thank you for the review!


1. If a Key has the metadata `unit/base` set to `hex` it will always be interpreted as a hex-value. The plugin will also produce an error,
if the value contained in such a Key does not start with `0x` (or `0X`).
2. If a Key has the metadata `type`, `unit/base` is not present, and the `type` value starts with `0x` (or `0X`) it will be interpreted as a hex-value.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"type value" is confusing. Maybe make two subpoints to make clear that two conditions need to be met (type integer, and the configuration value needs to start with 0x)

2. If a Key has the metadata `type`, `unit/base` is not present, and the `type` value starts with `0x` (or `0X`) it will be interpreted as a hex-value.
The following types are recognized as integer-types per default: `byte`, `short`, `unsigned_short`, `long`, `unsigned_long`,
`long_long`, `unsigned_long_long`
3. If forced conversion mode (`/force`, see below) is enabled all values starting with `0x` (or `0X`) are considered hex-values.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make clear that /force is a plugin configuration (and not meta data as the other two above)


## Configuration

When mounting a backend with the hexnumber plugin, a few parameters can be configured.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parameter -> setting (see https://www.libelektra.org/manpages/elektra-glossary).

And in the case of plugin configuration, its better to always use "plugin configuration".

starting with `0x` (or `0X`) into decimal before passing the value on to the rest of Elektra. This can be useful for importing a
configuration file that uses hexadecimal values into Elektra without writing a specification for the file.

NOTE: be careful when using this option, as strings like `0xg` might produce unwanted results. (see `man 3 strtoull` for details)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reference to strtoull unnecessarily exposes implementation details. And I think we should fail in such cases, everything else seems to be too dangerous.

@kodebach
Copy link
Member Author

everything should be fixed now

@markus2330
Copy link
Contributor

jenkins build all please

Copy link
Member

@sanssecours sanssecours left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems the plugin is leaking memory. To reproduce these problems locally you can enable the CMake option ENABLE_ASAN.

@@ -0,0 +1,10 @@
include (LibAddMacros)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please format this file with the script reformat-cmake.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for noting, we should enforce that with the buildserver. Isn't this just a matter of installing two tools?

Ideally the build server should directly tell that something like that is wrong.

@ingwinlu What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for noting, we should enforce that with the buildserver.

We already do that in the Travis Linux builds.

Isn't this just a matter of installing two tools?

Yes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

knock yourself out :D

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanssecours It seems you need to manually install libyaml as well as PyYAML for the reformat-cmake script to work. This should probably be reflected in our documentation. Especially because otherwise the script throws an error for each CMake file and also deletes the contents of all these files.

Copy link
Member

@sanssecours sanssecours May 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanssecours It seems you need to manually install libyaml as well as PyYAML for the reformat-cmake script to work.

As far as I know cmake-format has no dependencies:

pip show cmake-format
#> Name: cmake-format
#> Version: 0.3.6
#> Summary: Can format your listfiles so they don't look like crap
#> Home-page: https://github.com/cheshirekow/cmake_format
#> Author: Josh Bialkowski
#> Author-email: josh.bialkowski@gmail.com
#> License: UNKNOWN
#> Location: /Users/rene/.pyenv/versions/3.6.5/lib/python3.6/site-packages
#> Requires:
#> Required-by:

.

This should probably be reflected in our documentation. Especially because otherwise the script throws an error for each CMake file and also deletes the contents of all these files.

While reformat-cmake is certainly not perfect, it checks for the binaries cmake-format and sponge beforehand. If you did not install one of these tools it should just fail printing a (broken) error message. I tried this myself, first by uninstalling cmake-format and invoking the script. After that I uninstalled moreutils and ran the command again. At no point did the script change any file in my copy of the repo, at least as far as I can tell. Are you sure that the script deleted you CMake files? If so, can you please describe steps to reproduce the problem?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure that the script deleted you CMake files?

It did not delete the files themselves, just their contents. I think this is because stdout of cmake-format is piped back into the CMake file, but stdout is empty because of the error.

As far as I know cmake-format has no dependencies

I also looked at the cmake-format github page and could not find any mention of the dependency. But here the module yaml is imported, which is the cause of the error.

The problem could probably be solved by using the json or python format for the .cmake-format file, because these two modules are part of the python standard library.

If you still want me to, I can try to find out how to reproduce the error.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the detailed description. I opened pull request #2011, which should fix the problem by checking the exit code of cmake-format CMakeLists.txt before it updates any file.

The problem could probably be solved by using the json or python format for the .cmake-format file, because these two modules are part of the python standard library.

That might even be a better solution, than the one I came up with. Anyway, since I prefer the human readability of YAML over JSON or the Python config style, I did not change the format of the config file.

@markus2330
Copy link
Contributor

Please add elektraArrayGetStrings to tests/icheck.suppression (It is a newly added API).

I am not sure if some kind of iteration wouldn't be a better choice. Freeing the structure you return needs a loop (potential errors..). And it seems it cannot handle NULL-keys (binary Keys from which the value is a NULL pointer) because what you return needs to be NULL-terminated. And empty arrays might also be a problem?

What about something like elektraKsNextArrayEntry which simply forwards to the next array entry with the internal cursor? If it can be used within the high-level API, it would be ideal.

@kodebach
Copy link
Member Author

Please add elektraArrayGetStrings to tests/icheck.suppression (It is a newly added API).

I decided to remove the function again and use a KeySet together with elektraKsFilter instead. There should now be no more memory leaks.

@markus2330
Copy link
Contributor

jenkins build all please

@markus2330 markus2330 merged commit 36f2a2a into ElektraInitiative:master May 19, 2018
@markus2330
Copy link
Contributor

Thank you, great job: You could store the old string of the hex number into the internal metadata, then you would not lose the capitalization of hex numbers in the configuration files.

This was referenced Jun 27, 2018
@kodebach kodebach deleted the hexnumber branch August 25, 2019 11:53
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants