[Draft] Integrate arduino-dsmr-2 changes

.editorconfig

@@ -0,0 +1,9 @@
+root = true
+
+[*]
+end_of_line = lf
+trim_trailing_whitespace = true
+insert_final_newline = true
+charset = utf-8
+indent_style = space
+indent_size = 2

.github/workflows/main.yaml

@@ -0,0 +1,14 @@
+on: push
+
+jobs:
+  build-windows-msvc:
+    runs-on: windows-2022
+    steps:
+      - uses: actions/checkout@v4
+      - run: ./build-win.ps1
+  build-linux:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - run: sudo apt-get install -y clang
+      - run: ./build-linux.sh

.gitignore

@@ -0,0 +1,4 @@
+/build/
+/out/
+.idea/
+.vs/

CMakeLists.txt

@@ -0,0 +1,43 @@
+cmake_minimum_required (VERSION 3.20)
+project(arduino-dsmr-test LANGUAGES CXX)
+include(FetchContent)
+
+file(DOWNLOAD
+  https://github.com/doctest/doctest/releases/download/v2.4.12/doctest.h
+  ${CMAKE_BINARY_DIR}/doctest/doctest.h
+  EXPECTED_MD5 0671bbca9fb00cb19a168cffa89ac184)
+
+FetchContent_Populate(
+  mded_tls
+  URL https://github.com/Mbed-TLS/mbedtls/releases/download/mbedtls-3.6.4/mbedtls-3.6.4.tar.bz2
+  URL_HASH MD5=eb965a5bb8044bc43a49adb435fa72ee)
+set(ENABLE_PROGRAMS OFF CACHE INTERNAL "")
+set(ENABLE_TESTING OFF CACHE INTERNAL "")
+add_subdirectory(${mded_tls_SOURCE_DIR})
+
+FetchContent_Populate(
+  cmake_template
+  URL https://github.com/cpp-best-practices/cmake_template/archive/refs/heads/main.zip
+  URL_HASH MD5=c391b4c21eeabac1d5142bcf404c740c)
+include(${cmake_template_SOURCE_DIR}/cmake/CompilerWarnings.cmake)
+include(${cmake_template_SOURCE_DIR}/cmake/Sanitizers.cmake)
+
+file(GLOB_RECURSE arduino_dsmr_test_src_files CONFIGURE_DEPENDS "src/*.h" "src/*.cpp")
+add_executable(arduino_dsmr_test ${arduino_dsmr_test_src_files})
+target_include_directories(arduino_dsmr_test PRIVATE ${CMAKE_SOURCE_DIR}/src ${CMAKE_BINARY_DIR}/doctest)
+target_include_directories(arduino_dsmr_test SYSTEM PRIVATE $<TARGET_PROPERTY:mbedtls,INTERFACE_INCLUDE_DIRECTORIES>)
+target_compile_features(arduino_dsmr_test PRIVATE cxx_std_20)
+target_link_libraries(arduino_dsmr_test PRIVATE mbedtls)
+
+# enable warnings
+add_library(arduino_dsmr_test_warnings INTERFACE)
+myproject_set_project_warnings(arduino_dsmr_test_warnings ON "" "" "" "")
+if(CMAKE_CXX_COMPILER_ID MATCHES ".*Clang")
+  target_compile_options(arduino_dsmr_test_warnings INTERFACE -Wno-gnu-zero-variadic-macro-arguments)
+endif()
+target_link_libraries(arduino_dsmr_test PRIVATE arduino_dsmr_test_warnings)
+
+# enable sanitizers: address, leak, undefined behaviour
+add_library(arduino_dsmr_test_sanitizers INTERFACE)
+myproject_enable_sanitizers(arduino_dsmr_test_sanitizers ON ON ON OFF OFF)
+target_link_libraries(arduino_dsmr_test PRIVATE arduino_dsmr_test_sanitizers)

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2015 Matthijs Kooijman <matthijs@stdin.nl>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,246 +1,56 @@
-# Arduino Dutch Smart meter (DSMR) parser
-
-This is an Arduino library for interfacing with Dutch smart meters, through
-their P1 port. This library can take care of controlling the "request" pin,
-reading messages and parsing them.
-
-This code was written for Arduino, but most of the parsing code it is pretty
-generic C++ (except for the Arduino- and AVR-based string handling), so it
-should be possible to adapt for use outside of the Arduino environment.
-
-When using Arduino, version 1.6.6 or above is required because this
-library needs C++11 support which was enabled in that version.
-
-## Protocol
-
-Every smart meter in the Netherlands has to comply with the Dutch Smart
-Meter Requirements (DSMR). At the time of writing, DSMR 4.x is the
-current version. The DSMR 5.0 P1 specification is available and expected to
-be used in smart meters starting in 2016. This code should support both
-the 4.x and 5.0 specifications. 3.x meters might also work, but this has
-not been verified or tested (feedback welcome).
-
-The DSMR specifications can be found on [the site of Netbeheer
-Nederland][netbeheer], in particular on [this
-page][dossier-slimme-meter]. Of particular interest is the "P1 companion
-standard" that specifies the P1 port and protocol (though not very
-clearly). Specifications can also be found in the `specs` subdirectory
-of this repository (including some older versions that are no longer
-online, which is why I started collecting them here).
-
-[netbeheer]: http://www.netbeheernederland.nl
-[dossier-slimme-meter]: https://www.netbeheernederland.nl/dossiers/slimme-meter-15/documenten
-
-According to DSMR, every smart electricity meter needs to have a P1
-port. This is a [6p6c socket][6p6c] (commonly, but incorrectly referred
-to as RJ11 or RJ12). Telephone plugs will fit, provided that you have
-some that actually have 6 pins wired, or you have just 4 and do not need
-power from the P1 port.
-
-[6p6c]: http://en.wikipedia.org/wiki/Modular_connector#6P6C
-
-Pinouts and electrical specs can best be looked up in the spec (The 5.0
-version is the most clear in this regard, though not everything may
-apply to 4.x meters).
-
-Note that the message format for the P1 port is based on the IEC 62056-21
-"mode D" format. That spec is not available for free, though there seem
-to be [a version available on the net][iec62056-21]. DLMS also seems a
-related standard, but that apparently defines a binary format. It does
-seem all of these use "OBIS identifiers" and "COSEM data objects"
-(DLMS has [some lists of objects][objlists], of which 1001-7 seems to
-somewhat match th DSMR specs), to describe the various properties,
-though it's not entirely clear how all of these fit together. However,
-the DSMR spec has a complete, though sometimes confusing list of fields
-used.
-
-[iec62056-21]: https://www.ungelesen.net/protagWork/media/downloads/solar-steuerung/iec62056-21%7Bed1.0%7Den_.pdf
-[objlists]: http://www.dlms.com/documentation/listofstandardobiscodesandmaintenanceproces/index.html
-
-A typical P1 message looks something like this:
-
-    /KFM5KAIFA-METER
-
-    1-0:1.8.1(000671.578*kWh)
-    1-0:1.7.0(00.318*kW)
-    !1E1D
-
-This includes an identification header at the top, a checksum at the
-bottom, and one or more lines of data in the middle. This example is
-really stripped down, real messages will have more data in them.
-
-The first part of the line (e.g. `1-0:1.8.1`) is the (OBIS) id of the
-field, which defines the meaning and format of the rest of the line.
-
-## Parsing a message
-
-Unlike other solutions floating around (which typically do some pattern
-matching to extract the data they need), this code properly parses
-messages, verifying the checksum and really parses each line according
-to the specifications. This should make for more reliable parsing, and
-allows for useful parser error messages:
-
-    1-0:1.8.1(000671.578*XWh)
-                         ^
-    Error: Invalid unit
-
-    1-0:1.8.1(0006#71.578*kWh)
-                  ^
-    Error: Invalid number
-
-    !6F4A
-     ^
-    Checksum mismatch
-
-This library uses C++ templates extensively. This allows defining a
-custom datatype by listing the fields you are interested in, and then
-all necessary parsing will happen automatically. The code generated
-parses each line in the message in turn and for each line loops over the
-fields in the datatype to find one whose ID matches. If found, the value
-is parsed and stored into the corresponding field.
-
-As an example, consider we want to parse the identification and current
-power fields in the example message above. We define a datatype:
-
-    using MyData = ParsedData<
-      /* String */ identification,
-      /* FixedValue */ power_delivered
-    >;
-
-The syntax is a bit weird because of the template magic used, but the
-above essentially defines a struct with members for each field to be
-parsed. For each field, there is also an associated `xxx_present`
-member, which can be used to check whether the field was present in the
-parsed data (if it is false, the associated field contains uninitialized
-data). There is some extra stuff in the background, but the `MyData`
-can be used just like the below struct. It also takes up the same amount
-of space.
-
-    struct MyData {
-    	bool identification_present;
-    	String identification;
-    	bool power_delivered_present;
-    	FixedValue power_delivered;
-    };
-
-After this, call the parser. By passing our custom datatype defined
-above, the parser knows what fields to look for.
-
-      MyData data;
-      ParseResult<void> res = P1Parser::parse(&data, msg, lengthof(msg));
-
-Finally, we can check if the parsing was succesful and access the parsed
-values as members of `data`:
-
-      if (!res.err && res.all_present()) {
-        // Succesfully parsed, print results:
-        Serial.println(data.identification);
-        Serial.print(data.power_delivered.int_val());
-        Serial.println("W");
-      }
-
-In this case, we check whether parsing was successful, but also check
-that all defined fields were present in the parsed message (using the
-`all_present()` method), to prevent printing undefined values. If you
-want to support optional fields, you can use the `xxx_present` members
-for each field individually instead.
-
-Additionally, this template approach allows looping over all available
-fields in a generic way, for example to print the parse results with
-just a few lines of code. See the parse and read examples for how this
-works.
-
-Note that these examples contain the full list of supported fields,
-which causes parsing and printing code to be generated for all those
-fields, even if they are not present in the output you want to parse. It
-is recommended to limit the list of fields to just the ones that you
-need, to make the parsing and printing code smaller and faster.
-
-## Parsed value types
-
-Some values are parsed to an Arduino `String` value or C++ integer type,
-those should be fairly straightforward. There are three special types
-that need some explanation: `FixedValue` and `TimestampedFixedValue`.
-
-When looking at the DSMR P1 format, it defines a floating point format.
-It is described as `Fn(x,y)`, where `n` is the total number of (decimal)
-digits, of which at least `x` and at most `y` are behind the decimal
-separator (e.g. fractional digits).
-
-However, this floating point format is a lot more limited than the C
-`float` format. For one, it is decimal-based, not binary. Furthermore,
-the decimal separator doesn't float very far, the biggest value for `y`
-used is 3. Even more, it seems that for any given field, there is no
-actual floating involved, fields have `x` equal to `y`, so the number of
-fractional digits is fixed.
-
-Because of this, parsing into a `float` value isn't really useful (and
-on the Arduino, which doesn't have an FPU, very inefficient too). For
-this reason, we use the `FixedValue` type, which stores the value as an
-integer, in thousands of the original unit. This means that a value of
-1.234kWh is stored as 1234 (effectively the value has been translated to
-Wh).
-
-If you access the field directly, it will automatically be converted to
-`float`, keeping the original value. Alternatively, if an integer
-version is sufficient, you can call the `int_val()` method to get the
-integer version returned.
-
-    // Print as float, in kW
-    Serial.print(data.power_delivered);
-    // Print as integer, in W
-    Serial.print(data.power_delivered.int_val());
-
-Additionally there is a `TimestampedFixedValue` method, which works
-identically, but additionally has a `timestamp()` method which returns
-the timestamp sent along with the value.
-
-These timestamps are returned as a string, exactly as present in the P1
-message (YYMMDDhhmmssX, where X is S or W for summer- or wintertime).
-Parsing these into something like a UNIX timestamp is tricky (think
-leap years and seconds) and of limited use, so this just keeps the
-original format.
-
-## Connecting the P1 port
-
-The P1 port essentially consists of three parts:
-
-- A 5V power supply (this was not present in 3.x).
-- A serial TX pin. This sends meter data using 0/5V signalling, using
-  idle low. Note that this is the voltage level commonly referred to as
-  "TTL serial", but the polarity is reversed (more like RS232). This
-  port uses 115200 bps 8N1 (3.x and before used 9600 bps).
-- A request pin - 5V needs to be applied to this pin to start
-  generating output on the TX pin.
-
-To connect to an Arduino that has an unused hardware serial port (like
-an Arduino Mega, Leonardo or Micro), the signal has to inverted. This
-can be done using a dedicated inverter IC, or just a transistor and some
-resistors.
-
-It's also possible to do all of the serial reception, including the
-inverting, in software using Arduino's SoftwareSerial library. However,
-it seems there are still occasional reception errors when using
-SoftwareSerial.
-
-## Sub meters
-
-In addition to a smart electricity meter, there can be additional
-sub meters attached (e.g., gas, water, thermal and sub electricity
-meter). These can talk to the main meter using the (wired or wireless)
-MBUS protocol to regularly (hourly for 4.x, every 5 minutes for 5.0)
-send over their meter readings. Based on the configuration / connection,
-each of these subs gets an MBUS identifier (1-4).
-
-In the P1 message, this identifier is used as the second number in the
-OBIS identifiers for the fields for the sub. Currently, the code has
-the assignment from MBUS identifier to device type hardcoded in
-`fields.h`. For the most common configuration of an electricity meter with
-a single gas meter as sub, this works straight away. Other
-configurations might need changes to `fields.h` to work.
-
-## License
-
-All of the code and documentation in this library is licensed under the
-MIT license, with the exception of the examples, which are licensed
-under an even more liberal license.
+This is a fork of [matthijskooijman/arduino-dsmr](https://github.com/matthijskooijman/arduino-dsmr).
+The primary goal is to make the parser independent of the Arduino framework and usable on ESP32 with the ESP-IDF framework or any other platform.
+
+# Features
+* Combines all fixes from [matthijskooijman/arduino-dsmr](https://github.com/matthijskooijman/arduino-dsmr) and [glmnet/arduino-dsmr](https://github.com/glmnet/arduino-dsmr) including unmerged pull requests.
+* Added an extensive unit test suite
+* Small refactoring and code optimizations
+* Supported compilers: MSVC, GCC, Clang
+* Header-only library, no dependencies
+* Code can be used on any platform, not only embedded.
+
+# Differences from the original arduino-dsmr
+* Requires a C++20 compatible compiler.
+* [P1Reader](https://github.com/matthijskooijman/arduino-dsmr/blob/master/src/dsmr/reader.h) class is replaced with the [PacketAccumulator](https://github.com/PolarGoose/arduino-dsmr-2/blob/master/src/dsmr/packet_accumulator.h) class with a different interface to allow usage on any platform.
+* Added [EncryptedPacketAccumulator](https://github.com/PolarGoose/arduino-dsmr-2/blob/master/src/arduino-dsmr-2/encrypted_packet_accumulator.h) class to receive encrypted DSMR messages (like from "Luxembourg Smarty").
+
+# How to use
+## General usage
+The library is header-only. Add the `src/arduino-dsmr-2` folder to your project.<br>
+Note: `encrypted_packet_accumulator.h` header depends on [Mbed TLS](https://www.trustedfirmware.org/projects/mbed-tls/) library. It is already included in the `ESP-IDF` framework and can be easily added to any other platforms.
+
+## Usage from PlatformIO
+The library is available on the PlatformIO registry:<br>
+[PlatformIO arduino-dsmr-2](https://registry.platformio.org/libraries/polargoose/arduino-dsmr-2/installation)
+
+# Examples
+* How to use the parser
+  * [minimal_parse.ino](https://github.com/matthijskooijman/arduino-dsmr/blob/master/examples/minimal_parse/minimal_parse.ino)
+  * [parse.ino](https://github.com/matthijskooijman/arduino-dsmr/blob/master/examples/parse/parse.ino)
+* Complete example using PacketAccumulator
+  * [packet_accumulator_example_test.cpp](https://github.com/PolarGoose/arduino-dsmr-2/blob/master/src/test/packet_accumulator_example_test.cpp)
+* Example using EncryptedPacketAccumulator
+  * [encrypted_packet_accumulator_example_test.cpp](https://github.com/PolarGoose/arduino-dsmr-2/blob/master/src/test/encrypted_packet_accumulator_example_test.cpp)
+
+# History behind arduino-dsmr
+[matthijskooijman](https://github.com/matthijskooijman) is the original creator of this DSMR parser.
+[glmnet](https://github.com/glmnet) and [zuidwijk](https://github.com/zuidwijk) continued work on this parser in the fork [glmnet/arduino-dsmr](https://github.com/glmnet/arduino-dsmr). They used the parser to create [ESPHome DSMR](https://esphome.io/components/sensor/dsmr/) component.
+After that, the work on the `arduino-dsmr` parser stopped.
+Since then, some issues and unmerged pull requests have accumulated. Additionally, the dependency on the Arduino framework causes various issues for some ESP32 boards.
+This fork addresses the existing issues and makes the parser usable on any platform.
+
+## The reasons `arduino-dsmr-2` fork was created
+* Dependency on the Arduino framework limits the applicability of this parser. For example, it is not possible to use it on Linux.
+* The Arduino framework on ESP32 inflates the FW size and doesn't allow usage of the latest version of ESP-IDF.
+* Many pull requests and bug fixes needed to be integrated into the parser.
+* Lack of support for encrypted DSMR messages.
+
+# How to work with the code
+* You can open the code using any IDE that supports CMake.
+  * `build-windows.ps1` script needs `Visual Studio 2022` to be installed.
+  * `build-linux.sh` script needs `clang` to be installed.
+
+# References
+* [DSMR parser in Python](https://github.com/ndokter/dsmr_parser/tree/master) - alternative DSMR parser implementation in Python.
+* [SmartyReader](https://www.weigu.lu/microcontroller/smartyReader_P1/index.html) - open source hardware to communicate with P1 port.
+* [SmartyReader. Chapter "Encryption"](https://www.weigu.lu/microcontroller/smartyReader/index.html) - how the encrypted DSMR protocol works.