Unable to execute unit tests

[zaskar9]

The unit tests fail to execute when I execute the following commands on my MacBook (macOS Sonoma 14.3.1).

cmake .. -G "Unix Makefiles"
make test

I get the following error message.

usage: lit [-h] [--version] [-j N] [--config-prefix NAME] [-D NAME=VAL] [-q] [-s] [-v] [-vv] [-a] [-o PATH]
           [--no-progress-bar] [--show-excluded] [--show-skipped] [--show-unsupported] [--show-pass]
           [--show-flakypass] [--show-xfail] [--path PATH] [--vg] [--vg-leak] [--vg-arg ARG] [--time-tests]
           [--no-execute] [--xunit-xml-output XUNIT_XML_OUTPUT] [--resultdb-output RESULTDB_OUTPUT]
           [--time-trace-output TIME_TRACE_OUTPUT] [--timeout MAXINDIVIDUALTESTTIME] [--max-failures MAX_FAILURES]
           [--allow-empty-runs] [--ignore-fail] [--max-tests N] [--max-time N] [--order {lexical,random,smart}]
           [--shuffle] [-i] [--filter REGEX] [--filter-out REGEX] [--xfail LIST] [--xfail-not LIST] [--num-shards M]
           [--run-shard N] [--debug] [--show-suites] [--show-tests] [--show-used-features]
           TEST_PATH [TEST_PATH ...]
lit: error: argument -s/--succinct: ignored explicit argument ' -v'
make[3]: *** [test/CMakeFiles/test] Error 2
make[2]: *** [test/CMakeFiles/test.dir/all] Error 2
make[1]: *** [test/CMakeFiles/test.dir/rule] Error 2
make: *** [test] Error 2

I am using lit 17.0.6 and filecheck 17.0.6.

Furthermore, when I try to execute a single test, for example with lit -a codegen/array_1.mod, I also get an error that seems to indicate that the lib and include paths are concatenated the with ; (which is platform-specific to Windows). On Linux and macOS, these paths should be concatenated with :.

[zaskar9]

Exact same problem under Ubuntu 22.04.3 LTS (Linux 6.5.0-18).

[tenko]

I found the error in the CMake file where the string double quote marks are surrounding the arguments:

test/CMakeLists.txt
set(TEST_SUITE_LIT_FLAGS "-s -v" CACHE STRING "Flags used when running lit")
...
add_custom_target(test COMMAND ${TEST_SUITE_LIT} ${TEST_SUITE_LIT_FLAGS} .
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
USES_TERMINAL
)

This creates the command:

lit.exe "-s -v" .

Removing the double quote marks runs the tests correctly:

lit.exe -s -v .

I tried to search for a solution, but could not find anything with a quick google search.
You have any clue here?

The problem with the path separator I am thinking we need to pass a variable %psep in the lit setup script:

RUN: %oberon -I "%S%psep%inc" -L "%S%psep%lib" -l oberon --run %s | filecheck %s

[zaskar9]

There's actually a predefined variable %{pathsep} that is known to lit. With that and your comment I was able to fix both issues in the latest batch of bug fixes. I can now run the unit tests on both macOS and Linux, which is very helpful! Thanks!

I noticed that some of the unit tests need to be adapted as I changed some of the outputs that the compiler gives. I've been going through the Oberon 07 language report and started to "unsupport" certain functionality in accordance with the standard. I'm thinking of introducing flags later on to enable these features again as they are basically supported in the codegen. Also note that I changed how REAL numbers are printed to standard out by "porting" code by Gutknecht from Oberon V2 to C.

Finally, I would like to point out that some of the tests fail because they are supposed to fail. For example, it is not possible according the Oberon 90 or Oberon 07 grammar to have empty statements, e.g., ;; or ; at the end of a block. Also, empty procedures (and modules) are supported, but according the grammar, they do not have the BEGIN keyword as shown below.

PROCEDURE Empty;
END Empty;

Last but not least, function procedures always have parenthesis after the identifier (both in the declaration and when they are called), even if they have no parameters. Currently, the corresponding error message looks like the parser tripped up, but this issue is already fixed and there will be a better error message in an upcoming pull request.

[tenko]

I thinks it is good to use Oberon-07 report as a target first and then cover what is missing and
later add support for other versions.

The XDS compiler gives a warning for extra semicolons.
Error is no problem given the error message is explicit.

The empty procedure is probably an error from my side.

The procedure without parentheses could be from Oberon-2 (Will check this)

A large amount of code exists for the Oberon-2 and it has some nice features like bound procedures.
Component Pascal is also very close to Oberon-2.
I think it would be nice to add support for these variants also at a later stage.

It would probably be good to add extra types to the SYSTEM module for interface with external code
as the types could change depending on the language version (Oberon-07 does not have LONGINT or LONGREAL).
The XDS compiler does this and relaxes the argument type checking for external procedures.
The XDS documentation here could be worth checking out here.

With regards to printing of REAL for testing purposes it is maybe an idea to add the hex version as dividing/multiply by 16 does not introduce rounding errors and should be more stable.
Robust REAL parsing and printing is a tricky issue only recently been solved for round tripping values (Ref. Ryu and Dragonbox algorithms).

[zaskar9]

I spent a bit of time on getting more unit tests to pass. Your example brought some real oversights to the surface. In particular, I improved the scanner when it comes to correctly recognizing floating-point and integer constants. I've pushed the fixes with the latest pull request, which also includes a summary of what tests still fail.

With respect to Oberon language standards my plan is to initially support as many standards (Oberon-90, Oberon-2, and Oberon-07) as possible and ultimately have a compiler flag --std that lets one select the standard that should be used. To get there, I have to support the union of all features. That's why, for example, I have LONGINT and LONGREAL, even though they are not present in Oberon-07. The main reason for me to do that is that I think it's an interesting engineering problem to get the different language standards into one framework. Not sure, there's any practical use to it. 😀

[tenko]

Excellent.

I believe there are some features in Oberon-2 which makes it more practical for more general use.

Pure Oberon-07 is very suitable for embedded development as it forces a very conservative style of code similar to programming languages used on PLC devices.

I have a work in progress standard library project targeting the XDS Oberon-2 compiler which makes use of:

• Oberon-2 Type bound procedures
• Oberon-2 Dynamic allocation of array which allows for dynamic strings and data structures.
• XDS Extension for default arguments which improves the ergonomics of the API
• XDS Extension for variable argument passing which improves the ergonomics of the API (printing, logging etc)

Link to code : XDSStdLib
Link to documentation Doc

UI Example Link
DB Example Link

The XDS has some issues with 64bit support both in code and linking.
This will probably never be resolved i guess.

So with time I was hoping to port this over to the oberon-lang compiler when it matures.

[zaskar9]

So with time I was hoping to port this over to the oberon-lang compiler when it matures.

That's super scary! 🫣 I hope this project will get to this point eventually, even though I never planned for it to be used seriously.

[tenko]

The late professor N.Wirth was very wise and designed a minimal language which is perhaps possible to test to be complete watertight.
I firmly believe we enough unit tests it could definitely be usable to serious applications!

[zaskar9]

Well, "wise" is certainly one adjective one could use to describe him. He surely had a very clear vision of what he wanted to do and what he felt was the right of way of doing it. The rigid type systems of Wirth's languages that only allow for very controlled extensions certainly make preventing type-related run-time problems more easily possible than in other languages. However, this comes at the price of having no (developer-defined) overloading (e.g., ad-hoc polymorphism) and (almost) no subtype polymorphism. In contrast, as for preventing memory-related run-time problems, these languages are very weak and, one might argue, behind even C. The C standard at least specifies free as a means to release memory, while the Oberon language report mentions that as an option among others, including garbage collection. I thought of implementing built-in shared pointers following the example of Apple's Objective-C's Automatic Reference Counting (ARC). But that's way in the future. First, we need to get the basics working. 😎

By the way, I read in the documentation of your Oberon Standard Library that you are dissatisfied with the XDS Compiler due to its lack of 64bit support. I think this will not be an issue with oberon-lang as it has been developed and tested on both Intel x86-64 and aarch64 (M1/M3). I also frequently test it on macOS and Linux as well as, less frequently, Windows 11. However, I'm quite certain that it won't work correctly on 32bit platforms, except if there is some LLVM magic that I'm not yet aware of. There are many points in the code, where assumptions about the width of certain data types are made. Clearly, this would need to be addressed if the compiler was deployed in such settings.

Finally, since you mentioned Niklaus Wirth, you might be interested to read what I wrote after being invited by ETH Zurich to share my memories of him: https://inf.ethz.ch/news-and-events/editors-choice/niklaus-wirth-quotes-all.html. Mine is the second from the top.

[tenko]

With regards to practical use of the language I implemented a Dictionary/Set type with record extension and
some callbacks for handing the extended record. This is quite readable and works surprisingly good.
A simple benchmark against the C implementation shows that it runs at half the speed due to some extra indirections.
This was done with the XDS compiler from the late 90 and compared against a modern GCC.
A Oberon compiler with LLVM backend should do better.

This is found at ADTDictionary.ob2
And used in DataConfig.ob2

In my opinion Oberon-02 (or maybe Oberon-07 with some extensions) is very usable and still relevant.
The problem is that every language need to copy every fancy popular feature from other languages to stay relevant (e.g. the curse of the "market") and simpler languages like Oberon is falling behind here.

Not being an expert I see there is a flurry of interesting developments in many languages with regards to replacing traditional memory handling. Found this language intriguing Lobster .
(I know the author from my Amiga days where he developed a language called AmigaE. Really smart guy.).

With regards to working on 32bit platforms I believe the LLVM IR code should be able to work correctly in most cases.
Pointers are always 64bit in LLVM IR and at the actual code generation stage it will be optimized to the specific platform.
Pointers should therefore work OK. With the rest only testing will show.

I checked the links on Wirth. Thanks. Really nice to read.