Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MEN-6834: Make all JSON parsing case insensitive #1523

Conversation

lluiscampos
Copy link
Contributor

@lluiscampos lluiscampos commented Nov 29, 2023

Although the JSON standard defines the keys as case sensitive, the
golang standard library implements it as case insensitive. Therefore to
avoid regressions we need to make all our JSON parsing case insensitive.

To implement it, we define a custom nlohmann::basic_json with a custom
std::map that compares the keys as lowercase. See:

Signed-off-by: Lluis Campos <lluis.campos@northern.tech>
@mender-test-bot
Copy link

@lluiscampos, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".


my commands and options

You can trigger a pipeline on multiple prs with:

  • mentioning me and start pipeline --pr mender/127 --pr mender-connect/255

You can start a fast pipeline, disabling full integration tests with:

  • mentioning me and start pipeline --fast

You can trigger GitHub->GitLab branch sync with:

  • mentioning me and sync

You can cherry pick to a given branch or branches with:

  • mentioning me and:
 cherry-pick to:
 * 1.0.x
 * 2.0.x

@lluiscampos lluiscampos force-pushed the MEN-6834-config-parsing-case-insensitive branch from 6f0a11b to e31bd2c Compare November 29, 2023 14:59
template <class Key, class T, class IgnoredLess, class Allocator = allocator<pair<const Key, T>>>
class CaseInsensitiveMap : public map<const Key, T, CaseInsensitiveLess, Allocator> {
public:
using CaseInsensitiveMapType = map<const Key, T, CaseInsensitiveLess, Allocator>;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I tried (the hard way) with unordered_map as I think it would be a better fit (plus we could reuse a tiny piece of code from the HTTP) but although the documentation seems to indicate that it should work, it does not work. See here or here.

@lluiscampos
Copy link
Contributor Author

@mender-test-bot start pipeline

@mender-test-bot
Copy link

Hello 😸 I created a pipeline for you here: Pipeline-1089545281

Build Configuration Matrix

Key Value
AUDITLOGS_REV master
BUILD_BEAGLEBONEBLACK true
BUILD_CLIENT true
BUILD_CPP_CLIENT true
BUILD_QEMUX86_64_BIOS_GRUB true
BUILD_QEMUX86_64_BIOS_GRUB_GPT true
BUILD_QEMUX86_64_UEFI_GRUB true
BUILD_VEXPRESS_QEMU true
BUILD_VEXPRESS_QEMU_FLASH true
BUILD_VEXPRESS_QEMU_UBOOT_UEFI_GRUB true
CREATE_ARTIFACT_WORKER_REV master
DEPLOYMENTS_ENTERPRISE_REV master
DEPLOYMENTS_REV master
DEVICEAUTH_ENTERPRISE_REV master
DEVICEAUTH_REV master
DEVICECONFIG_REV master
DEVICECONNECT_REV master
DEVICEMONITOR_REV master
GENERATE_DELTA_WORKER_REV master
GUI_REV master
INTEGRATION_REV feature-c++-client
INVENTORY_ENTERPRISE_REV master
INVENTORY_REV master
IOT_MANAGER_REV master
MENDER_ARTIFACT_REV master
MENDER_BINARY_DELTA_REV master
MENDER_CLI_REV master
MENDER_CONFIGURE_MODULE_REV master
MENDER_CONNECT_REV master
MENDER_CONVERT_REV master
MENDER_GATEWAY_REV master
MENDER_REV pull/1523/head
MENDER_SETUP_REV master
MENDER_SNAPSHOT_REV master
META_MENDER_REV feature-c++-client
META_OPENEMBEDDED_REV kirkstone
META_RASPBERRYPI_REV kirkstone
MONITOR_CLIENT_REV master
MTLS_AMBASSADOR_REV master
POKY_REV kirkstone
RUN_BACKEND_INTEGRATION_TESTS false
RUN_INTEGRATION_TESTS true
TENANTADM_REV master
TEST_QEMUX86_64_BIOS_GRUB true
TEST_QEMUX86_64_BIOS_GRUB_GPT true
TEST_QEMUX86_64_UEFI_GRUB true
TEST_VEXPRESS_QEMU true
TEST_VEXPRESS_QEMU_FLASH true
TEST_VEXPRESS_QEMU_UBOOT_UEFI_GRUB true
USERADM_ENTERPRISE_REV master
USERADM_REV master
WORKFLOWS_ENTERPRISE_REV master
WORKFLOWS_REV master

Copy link
Member

@kacf kacf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

os << R"({
"artifactverifykey": "ArtifactVerifyKey_value",
"deviceTypeFile": "DeviceTypeFile_value",
"SERVERURL": "ServerURL_value"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should have a separate test where you have the same key with different casing? I'm a little bit interested to see how it handles it, especially when compared to having two completely identical keys.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It behaves the same with two identical keys than with same key with different casing: the entry in the map gets overridden and only the last one stands.

I'll add a test, thanks for the suggestion.

Although the JSON standard defines the keys as case sensitive, the
golang standard library implements it as case insensitive. Therefore to
avoid regressions we need to make all our JSON parsing case insensitive.

To implement it, we define a custom `nlohmann::basic_json` with a custom
`std::map` that compares the keys as lowercase. See:
* https://json.nlohmann.me/api/basic_json/basic_json/
* https://json.nlohmann.me/api/basic_json/object_t/

Ticket: MEN-6834
Changelog: None

Signed-off-by: Lluis Campos <lluis.campos@northern.tech>
@lluiscampos lluiscampos force-pushed the MEN-6834-config-parsing-case-insensitive branch from e31bd2c to cb118d4 Compare November 30, 2023 07:38
@lluiscampos
Copy link
Contributor Author

The last force-push only added a unit-test, not running again the full pipeline. See the last pipeline here.

@lluiscampos lluiscampos merged commit b21c212 into mendersoftware:feature-c++-client Nov 30, 2023
2 checks passed
@lluiscampos lluiscampos deleted the MEN-6834-config-parsing-case-insensitive branch November 30, 2023 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants