SLP2

[urbasus]

https://tools.ietf.org/html/rfc2608

[urbasus]

I'll try implement SLP2, will post significant progress here (if any).

[senier]

I'm curious to see the result. As we don't have good documentation, yet, this ticket would also be the right place to start a discussion on how to specify certain aspects of SLP2.

[urbasus]

I made an enum member and type that had the same name but different case. The parser could not handle it. Is this by design? Not seeing this property mentioned in any docs.

Snippet:

   type Function_Id is (
      SERVICE_REQUEST => 1,
      SERVICE_REPLY => 2,
      SERVICE_REGISTRATION => 3,
      SERVICE_DEREGISTER => 4,
      SERVICE_ACKNOWLEDGE => 5,
      ATTRIBUTE_REQUEST => 6,
      ATTRIBUTE_REPLY => 7,
      DIRECTORY_AGENT_ADVERTISEMENT => 8,
      SERVICE_TYPE_REQUEST => 9,
      SERVICE_TYPE_REPLY => 10,
      SERVICE_AGENT_ADVERTISEMENT => 11
   ) with Size => 8;

  -- message header...

   -- message bodies/payloads
   type Service_Request is
      message
         Length_Of_Previous_Responder_List : Length_Of_Previous_Responder_List
            then Previous_Responder_List
               with Size => Length_Of_Previous_Responder_List * 8;
         Previous_Responder_List : Opaque;
      end message;

   -- Collision between Service_Request and SERVICE_REQUEST
   for SLPv2 use (Payload => Service_Request)
      if Function_Id = SERVICE_REQUEST;

Error message:

$ rflx generate ../slpv2.rflx
Parsing ../slpv2.rflx
Processing SLPv2
../slpv2.rflx:23:7: model: error: literal "SERVICE_REQUEST" conflicts with type declaration
../slpv2.rflx:68:4: model: info: conflicting type "SLPv2::Service_Request"

Adding _Type suffix to types is the obvious workaround. Recall that being a popular convention.

[senier]

You're right, identifiers in RecordFlux are case-insensitive (making SERVICE_REQUEST the same thing as Service_Request) and we don't mention that in the language reference. I guess we are too used to Ada identifiers being case-insensitive. I added this to AdaCore/RecordFlux#342 so we don't forget to document it.

Regarding the _Type suffix: I used to do it this way, but at some point felt that this is superfluous and impairs readability. Today I try to avoid it. In your above example I'd probably prefix all enum elements with something like FI_ (which also avoids the issue of conflicting literals of different enums, which happened to me frequently for some protocols). But of course that's a matter of taste.

[urbasus]

I hit an issue I'm not sure how to solve. In RFC 2608, section 9.1 Service Location Protocol Extensions, we read:

The three byte offset to next extension indicates the position of the
next extension as offset from the beginning of the SLP message.

The offset value is 0 if there are no extensions following the
current extension.

If the offset is 0, the length of the current Extension Data is
determined by subtracting total length of the SLP message as given in
the SLP message header minus the offset of the current extension.

Basically a linked list of sorts. How would I express this?

[senier]

I hit an issue I'm not sure how to solve. In RFC 2608, section 9.1 Service Location Protocol Extensions, we read:
[...]
Basically a linked list of sorts. How would I express this?

Sometimes I wonder what went wrong in the childhoods of those protocol designers ;)

The mechanism cannot be expressed with RecordFlux as is. I'd call this a feature, though, as absolute offsets to define the start of another extension is dangerous for (at least) three reasons:

1. You may have holes in your message (what do you fill them with when parsing, what do you initialize them with when creating a message). Implicit holes in a message are prevented by RecordFlux' verifier also for regular fields. The start of a field always has to follow the end of another field. Holes need to be made explicit by adding reserved fields.
2. Even more problematic is that fact that extensions may overlap (with each other or other fields in a message) or the offset may even point outside the message. Depending on the implementation this may enable TOCTOU attacks, memory corruption, data exfiltration and more.
3. You could create cycles which may result in DoS or resource exhaustion in the parser.

I'm inclined to create some specially crafted packets and try throwing them at an implementation. However, I have no idea what SLP2 is even used for and where it is implemented. If you know any accessible implementations, I'd be interested.

I don't think following the RFC by the letter is the right approach here. Do you know what implementations are actually doing? Do you have captured traces? I could imagine that this is actually just a list with the offset always pointing to the next bit after the current extension. This could be modeled by a sequence of messages (with the offset being ignored - consistency of the offset would need to be checked manually later).

If the linked list is legit and indeed used, we may have to think about a way to safely express it (and generate provable code out of it which will for sure be a major challenge).

[urbasus]

Suspected as much. Yes, the extension mechanism has many complicated properties. Perhaps this outcome of trying to specify it in RecordFlux can provide insights for future specifications/revisions (although admittedly SLP is stale and mostly superseded by SSDP).

I think pushing it to implementation is the right thing to do. The list of extensions is a continuous block of memory whose size is easily derived from the header - no problem there. And besides, the only mandatory extension - Required_Attribute_Missing - is (I think) just nicer error handling for more elaborate service providers, which I think is fine to omit.

There is but one major implementation, OpenSLP. It is implemented entirely in C. I decided to try implementing an alternative in Ada, in part because of a desire to learn more Ada and SPARK, in part because I'd like a minimal, reliable implementation for lab/embedded environments (which given the open issues of OpenSLP seem unlikely). That being said OpenSLP has a command line tool that is very useful for testing an alternative implementation against a reference. Wireshark even knows how to read the fields.

For what I know SLP is nowadays mostly used to locate printers on a network, though I do see potential in lab/embedded networks as means to more easily configure and modify the network. It has similarities to SSDP, which is part of UPnP. In a nutshell a client makes a broadcast/multicast request for a service matching "patterns" (is color printer, DPI this or that, located in facility X floor 2), and service providers then respond with URL (including port).

Know I do this for my own pleasure. Happy as long as I learn something new.

[urbasus]

Having issues running test.

pytest complains about -n<num> being an invalid argument:

$ make test_python
python3 -m pytest -n 4 -vv
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: -n

Not familiar with pytest. Could not find reference mentioning a -n flag.

2. Having removed -n<num> from Makefile, gprbuild step fails:

$ make test_python
python3 -m pytest -vv
=================================== test session starts ===================================
platform linux -- Python 3.7.3, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: ...
collected 12 items

tests/test_specs.py::test_spec[ethernet.rflx] FAILED                                [  8%]
tests/test_specs.py::test_spec[http_2.rflx] ^C

======================================== FAILURES =========================================
________________________________ test_spec[ethernet.rflx] _________________________________

spec = 'ethernet.rflx'
tmp_path = PosixPath('/tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0')

    @pytest.mark.parametrize("spec", glob.glob("*.rflx"))
    def test_spec(spec: str, tmp_path: pathlib.Path) -> None:
        subprocess.run(["rflx", "generate", "-d", tmp_path, spec], check=True)
>       subprocess.run(["gprbuild", "-U"], check=True, cwd=tmp_path)

tests/test_specs.py:11:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = None, check = True
popenargs = (['gprbuild', '-U'],)
kwargs = {'cwd': PosixPath('/tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0')}
process = <subprocess.Popen object at 0x7fa25ee3a390>, stdout = None, stderr = None
retcode = 4

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.

        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.

        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.

        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.

        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.

        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if 'stdin' in kwargs:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE

        if capture_output:
            if ('stdout' in kwargs) or ('stderr' in kwargs):
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE

        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired:
                process.kill()
                stdout, stderr = process.communicate()
                raise TimeoutExpired(process.args, timeout, output=stdout,
                                     stderr=stderr)
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
                raise CalledProcessError(retcode, process.args,
>                                        output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['gprbuild', '-U']' returned non-zero exit status 4.

/usr/lib/python3.7/subprocess.py:487: CalledProcessError
---------------------------------- Captured stdout call -----------------------------------
using project file /usr/share/gpr/_default.gpr
Compile
   [Ada]          rflx-rflx_message_sequence.adb
---------------------------------- Captured stderr call -----------------------------------
Parsing ethernet.rflx
Processing Ethernet
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-ethernet.ads
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-ethernet-generic_frame.ads
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-ethernet-generic_frame.adb
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-ethernet-frame.ads
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_arithmetic.ads
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_builtin_types-conversions.ads
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_builtin_types.ads
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_generic_types.ads
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_message_sequence.ads
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_scalar_sequence.ads
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_types.ads
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_arithmetic.adb
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_generic_types.adb
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_message_sequence.adb
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_scalar_sequence.adb
Creating /tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx.ads
+===========================GNAT BUG DETECTED==============================+
| 8.3.0 (x86_64-linux-gnu) Program_Error lib-writ.adb:763 explicit raise   |
| Error detected at rflx-rflx_generic_types.ads:30:80 [rflx-rflx_message_sequence.ads:13:4]|
| Please submit a bug report; see https://gcc.gnu.org/bugs/ .              |
| Use a subject line meaningful to you and us to track the bug.            |
| Include the entire contents of this bug box in the report.               |
| Include the exact command that you entered.                              |
| Also include sources listed below.                                       |
+==========================================================================+

Please include these source files with error report
Note that list may not be accurate in some cases,
so please double check that the problem can still
be reproduced with the set of files listed.
Consider also -gnatd.n switch (see debug.adb).

/tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_message_sequence.adb
/tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_message_sequence.ads
/tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx.ads
/tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_generic_types.ads
/tmp/pytest-of-andreas/pytest-3/test_spec_ethernet_rflx_0/rflx-rflx_arithmetic.ads

compilation abandoned
gprbuild: *** compilation phase failed
================================= short test summary info =================================
FAILED tests/test_specs.py::test_spec[ethernet.rflx] - subprocess.CalledProcessError: Co...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/usr/lib/python3.7/subprocess.py:1582: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
==================================== 1 failed in 2.52s ====================================
make: *** [Makefile:33: test_python] Error 2

Offending lines:

12: generic
13:    with package Types is new RFLX.RFLX_Generic_Types (<>);

Probably a problem with my machine. Grateful for pointers on how to proceed.

[urbasus]

Several Length fields in the SLPv2 protocol hold the length of the entire message, not just a field. To constrain field size properly in these cases the field offset is subtracted:

   -- 9.2. Authentication Blocks
   type Authentication_Block_Authentication_Block_Length is range 10 .. 2**16-1 with Size => 16;
   type Authentication_Block is
      message
         Block_structure_Descriptor : U16;
         Authentication_Block_Length : Authentication_Block_Authentication_Block_Length;
         Authentication_Block_Timestamp : U32;
         SLP_SPI_String_Length : U16;
         SLP_SPI_String : Opaque with Size => SLP_SPI_String_Length * 8;
         Structured_Authentication_Block : Opaque
            with Size => (Authentication_Block_Length - 10 + SLP_SPI_String_Length) * 8;
      end message;

The use of magic numbers (10) is unfortunate. One way would be to sum the size of preceding fields, though this is almost as error prone and hardly more readable.

Is there a way to use field offsets in size constraints?

I think Structured_Authentication_Block'Position would do what I want, but rflx does not seem able to parse it as of current.

[senier]

Seems like you're having 2 unrelated issues here.

pytest complains about -n<num> being an invalid argument:

Pytest realizes a lot of functionality through plugins. Even parallelization is done using the xdist plugin, which must be installed. pip install pytest-xdist should do the trick.

$ make test_python
[snip]
+===========================GNAT BUG DETECTED==============================+
| 8.3.0 (x86_64-linux-gnu) Program_Error lib-writ.adb:763 explicit raise |
| Error detected at rflx-rflx_generic_types.ads:30:80 [rflx-rflx_message_sequence.ads:13:4]|
| Please submit a bug report; see https://gcc.gnu.org/bugs/ . |
| Use a subject line meaningful to you and us to track the bug. |
| Include the entire contents of this bug box in the report. |
| Include the exact command that you entered. |
| Also include sources listed below. |
+==========================================================================+
[snip]
Probably a problem with my machine. Grateful for pointers on how to proceed.

The problem is the compiler crash above, which is probably a known problem. I suspect you're using the FSF GNAT of your distribution. We only support GNAT Community Edition 2020 due to various compiler issues that our generated code triggers in older compilers.

[senier]

Several Length fields in the SLPv2 protocol hold the length of the entire message, not just a field. To constrain field size properly in these cases the field offset is subtracted:

   -- 9.2. Authentication Blocks
   type Authentication_Block_Authentication_Block_Length is range 10 .. 2**16-1 with Size => 16;
   type Authentication_Block is
      message
         Block_structure_Descriptor : U16;
         Authentication_Block_Length : Authentication_Block_Authentication_Block_Length;
         Authentication_Block_Timestamp : U32;
         SLP_SPI_String_Length : U16;
         SLP_SPI_String : Opaque with Size => SLP_SPI_String_Length * 8;
         Structured_Authentication_Block : Opaque
            with Size => (Authentication_Block_Length - 10 + SLP_SPI_String_Length) * 8;
      end message;

The use of magic numbers (10) is unfortunate. One way would be to sum the size of preceding fields, though this is almost as error prone and hardly more readable.

Is there a way to use field offsets in size constraints?

I think Structured_Authentication_Block'Position would do what I want, but rflx does not seem able to parse it as of current.

You can use the 'First and 'Last attributes of message fields to achieve that:

         Structured_Authentication_Block : Opaque
            with Size => 8 * Authentication_Block_Length
                         - (SLP_SPI_String'Last - Block_Structure_Descriptor'First + 1);

(RecordFlux will throw an error regarding a potentially negative size of Structured_Authentication_Block for this model. This is correct - you'll probably need to add some constraints to assure that Authentication_Block_Length is big enough to hold the fixed fields, SLP_SPI_String and Structured_Authentication_Block).

[urbasus]

The problem is the compiler crash above, which is probably a known problem. I suspect you're using the FSF GNAT of your distribution. We only support GNAT Community Edition 2020 due to various compiler issues that our generated code triggers in older compilers.

Huh. I couldn't get rflx to start on Windows so I assumed the project was Linux only. Existence of GNU Makefiles strengthened that assumption. But GNAT CE is clearly Windows only. Cygwin?

Suppose I could punch a hole through WSL to use Windows GNAT install instead of WSL, but something tells me this is not how the development environment is supposed to be setup.

[senier]

Huh. I couldn't get rflx to start on Windows so I assumed the project was Linux only. Existence of GNU Makefiles strengthened that assumption. But GNAT CE is clearly Windows only. Cygwin?

It's true that RecordFlux is supported on Linux only for the time being (even though I see not fundamental obstacles to make it work on Windows or OS X). GNAT CE is available for Windows, Linux and OS X - if you go to more options on the download page, you can select the platform x86 GNU Linux (64 bits).

Suppose I could punch a hole through WSL to use Windows GNAT install instead of WSL, but something tells me this is not how the development environment is supposed to be setup.

That sounds scary ;)

[urbasus]

Huh. I couldn't get rflx to start on Windows so I assumed the project was Linux only. Existence of GNU Makefiles strengthened that assumption. But GNAT CE is clearly Windows only. Cygwin?

It's true that RecordFlux is supported on Linux only for the time being (even though I see not fundamental obstacles to make it work on Windows or OS X). GNAT CE is available for Windows, Linux and OS X - if you go to more options on the download page, you can select the platform x86 GNU Linux (64 bits).

Not the first time I missed "more options" on that very same download page.

Suppose I could punch a hole through WSL to use Windows GNAT install instead of WSL, but something tells me this is not how the development environment is supposed to be setup.

That sounds scary ;)

Well, it worked! A link from gprbuild to gprbuild.exe, prefix PATH with /mnt/c/... and voila, all specs but mine pass the tests (compiling)! I installed the Linux environment just in case the problem was still something in my "unconventional" environment, but I get the same error. Spec fail to compile with this message:

rflx-slpv2-service_registration.ads:16:140: unmatched actual in instantiation of "Generic_Service_Registration" declared at rflx-slpv2-generic_service_registration.ads:17

Offending line:

package RFLX.SLPv2.Service_Registration is new RFLX.SLPv2.Generic_Service_Registration (RFLX.RFLX_Types, RFLX.SLPv2.Authentication_Blocks, RFLX.SLPv2.Authentication_Blocks);

Relevant parts of the spec (see PR for full source):

   -- 9.2. Authentication Blocks
   type Authentication_Block_Authentication_Block_Length is range 10 .. 2**16-1 with Size => 16;
   type Authentication_Block is
      message
         Block_structure_Descriptor : U16;
         Authentication_Block_Length : Authentication_Block_Authentication_Block_Length;
         Authentication_Block_Timestamp : U32;
         SLP_SPI_String_Length : U16;
         SLP_SPI_String : Opaque with Size => SLP_SPI_String_Length * 8;
         Structured_Authentication_Block : Opaque
            with Size => (Authentication_Block_Length - (Block_structure_Descriptor'First - SLP_SPI_String'Last + 1)) * 8;
      end message;
   type Authentication_Blocks is array of Authentication_Block;

   -- 4.3. URL Entries
   type URL_Entry_Reserved is range 0 .. 0 with Size => 8;
   type URL_Entry is
      message
         Reserved : URL_Entry_Reserved;
         Lifetime : U16;
         URL_Length : U16;
         URL : Opaque with Size => URL_Length * 8;
         Number_Of_URL_Authentications : U8;
         Authentication_Blocks : Authentication_Blocks
            with Size => Number_Of_URL_Authentications * 8;
      end message;
   type URL_Entries is array of URL_Entry;

   -- 8.2. Service Reply
   type Service_Reply is
      message
         Error_Code : Error;
         URL_Entry_Count : U16;
         URL_Entries : URL_Entries with Size => URL_Entry_Count * 8;
      end message;
   for SLPv2 use (Payload => Service_Reply)
      if Function_ID = FI_SERVICE_REPLY;

   -- 8.3. Service Registration
   type Service_Registration is
      message
         URL_Entry : URL_Entry;
         Length_Of_Service_Type_String : U16;
         Service_Type : Opaque with Size => Length_Of_Service_Type_String * 8;
         Length_Of_Scope_List : U16;
         Scope_List : Opaque with Size => Length_Of_Scope_List * 8;
         Length_Of_Attribute_List_String : U16;
         Attribute_List : Opaque with Size => Length_Of_Attribute_List_String * 8;
         Number_Of_Attribute_Authentication_Blocks : U8;
         Attribute_Authentication_Blocks : Authentication_Blocks
            with Size => Number_Of_Attribute_Authentication_Blocks * 8;
      end message;
   for SLPv2 use (Payload => Service_Registration)
      if Function_ID = FI_SERVICE_REGISTRATION;

Sound familiar? Google-searches turn up almost blank on the general parts of the message.

[senier]

Well, it worked! A link from gprbuild to gprbuild.exe, prefix PATH with /mnt/c/... and voila, all specs but mine pass the tests (compiling)!

Wow, this is interesting to know! Congratulations for being the first Windows user of RecordFlux (for whatever it's worth ;)

I installed the Linux environment just in case the problem was still something in my "unconventional" environment, but I get the same error. Spec fail to compile with this message:

This is a bug. I created a reproducer and opened AdaCore/RecordFlux#536 - we'll look into it.

[senier]

This is a bug. I created a reproducer and opened Componolit/RecordFlux#536 - we'll look into it.

Please try AdaCore/RecordFlux@79bb58f.

[urbasus]

Please try Componolit/RecordFlux@79bb58f.

It worked!