GnuCOBOL test bed

[GitMensch]

In general I'm still trying to get a working GnuCOBOL with OrangeC - which is my main interest for now as it provides a nice testsuite.
Providing you with my local testbed (including all "special" headers, libraries and dll's needed) looks to be the easiest way to give you the option to reproduce different issues reported.

I've just upgraded to the last CI build which shows green, that is currently 1.0.397 - binaries: https://ci.appveyor.com/api/buildjobs/qgn0qthirt1a80qg/artifacts/dist%2Focc60371e.zip - and I can reproduce the following issues.

Just:

• unpack
• adjust testbed\build_windows\defaults.h to use your path
• start ocide
• open testbed\build_windows\ocide\GnuCOBOL.cwa
• click on Rebuild all
• fixed: see oclink access violation access violation in olink.exe #165 (ignore for now)
• click on "debug -> yes, rebuild"
• right click on cobcrun, click start
• fixed: see ocide.exe not responding as noted in Fixing the reporting error to microsoft window #161 (comment) - as soon as this is fixed go back to the access violation in olink.exe #165 issue
• switch to Release version
• adjust libcob/common.h to read (just look for DECLNORET)

#if defined	(_MSC_VER) || defined (__ORANGEC__) || \
   (defined (__BORLANDC__) && defined (_WIN32))
#define	DECLNORET	__declspec(noreturn)

• click on Rebuild all
• works here, 82 warnings - fixed: including many "105" because of FR: add support for __declspec(noreturn) #136
• start a cmd.exe, adjust PATH to include orangec/bin
• cd to testbed\build_windows\ocide\WIN32\Release
• do cobc testme.c -x -vv (the compiled "cobc" calls occ and creates a working executable (it needed 40 days to come this far ;-) and then runs into a SIGSEGV (no other of the tested compilers [should be more than 6 in a multitude of versions and OS and library combinations] has this result) - fixed
• do cobcrun --info, fixed: raising another SIGSEGV (I assume this comes from the RTL on/near program end)

bonus:

• start mingw and go to testbed/build_orangec
• adjust testbed/build_orangec/source_configure to use your path
• run source_configure (passes here)
• seems to be fixed: run omake - and see the issue mentioned at omake is not GNU make compatible, raises Assertion #56 (comment) (fix later or directly)
• run make instead, everything compiles, fixed: linking fails because occ assumes .a and .lib to be C source files - FR FR: Is there any option to use .a and .lib as alias names? #154

I may go on with this list later, but for now:
gnucobol-testbed.zip 17MB

[LADSoft]

handing me this was a good idea; I'm able to fix a lot of stuff beyond what you talked about here.

A couple of things are probably going to come out of this: first I now see the need for a 'modules' window and second when #line is used maybe the debugger should index the original source file...

[LADSoft]

I did a lot of stuff with this, now you can debug reasonably and the files run after you build them :) Still working on the msys build though...

[LADSoft]

as of right now cobc compiles in the IDE and runs; it compiles also with the autoconf scripts but crashes when you run it.

[GitMensch]

I'm starting to run the testsuite with the generated programs (currently only possible with the IDE generated one because of the libtool wrapper executables chrashing as noted in #56).

Test results so far (I won't go on for now because of #203 resulting in many errors and the crash with #207 is stopping actual run tests):

• missing binary mode for stdout/stderr _setmode with _O_BINARY doesn't work (\n is translated to \r\n) #203
• missing pass of output names given to occ to sub-programs (only for oasm?) occ doesn't pass output file name to oasm #204
• wrong access or perror access to missing file results in "Permission denied" instead of "No such file or directory" #205
• output name handling strange output name handling (occ/olink producing multiple output files) #206
• long running compilation and Access Violation of NIST test runner, if not compiled with debug long running compilation with "1 Diagnostic" ... and an access violation in the resulting binary #207
• the creation of a file seems to result in ENOENT creation of a file seems to result in ENOENT #208

[LADSoft]

i've fixed all these except the binary mode issue and the broken NIST test runner. I was thinking the binary mode issue may be related to #130 because I could have sworn the behavior was wrong at one point but couldn't duplicate it later. I'll have to write a little test program and really exercise it.

[GitMensch]

Please reopen as not all issues are fixed yet.

[LADSoft]

Sorry I guess I got close-happy, I didn't intend to close this though :)

[GitMensch]

GCC can directly link against a dll and will create its import definitions as needed. Just use gcc test.c path/to/ibcob-4.dll Instead of gcc test.c -lcob Did I understand your question correctly? And wasn't there a FR for occ to be able to do the same?

[LADSoft]

yeah that helped, also had to have a version of GCC that would target 32 bits

I believe we did add it to OCC at some point now that you mention it...

you said in reference to test 905:

    I'll check the EAGAIN and drop a note later.

any news?

Meanwhile I made progress with the shared runtime library, got cobc to compile with it in a way that sorta works. It is a bit flaky though, need to iron that out...

[GitMensch]

re test 905 EAGAIN - on some systems open may return it if the file is locked, but that isn't even mentioned in its manpage.

It may be returned from fcntl (but I guess that's not implemented), otherwise on _WIN32 instead of EAGAIN the kernel LockFileEx function is called and if it can't create the specified lock then the same expected return code of io/status 61 is raised as it otherwise is with EAGAIN - and I think this is what the test does.

Summary:

• program 1 creating a file, then reopening it (input only); calling LockFileEx with LOCKFILE_FAIL_IMMEDIATELY (that should create a read lock)
• program 2 opening that file (inpu-output; calling LockFileEx with LOCKFILE_FAIL_IMMEDIATELY | LOCKFILE_EXCLUSIVE_LOCK (that should create a read-write lock, that should fail because of the existing read lock)

Before inspecting this in detail it may be useful to check if the tests pass with plain MSVC (which should take the same route as OrangeC in this case, GCC should take the fcntl route).

[LADSoft]

I can't completely test with MSVC without compiling libcob with MSVC, not something I want to do. But at one point I did write C language programs that did exactly what I think COBC is doing, and didn't get an EAGAIN out of MSVC... about then is when I asked the question lol...

[GitMensch]

ok, then just check that it gets to LockFileEx in both programs and check that this gets some reasonable result

[GitMensch]

Thanks for the pushes.

I'm evaluating whether we can have another release at this point.... main problem with a release being I'd been having ongoing problems with my alternate branches just randomly crashing during builds. But I may have fixed that now... have put everything on master and assuming the current test succeeds I will push it to see what appveyor does with it.

Originally posted by @LADSoft in #603 (comment)

Does that mean that the following is solved now, too?

yeah when I tried using the dynamic version of the runtime library i fell flat on my face... it crashed. So I punted and went back to using the static version but patching COBC to get around the issue. But some of the tests apparently do printfs in the main program (not by using LIBCOB) so they still have a problem...

I intend to get back to figuring out why the runtime library doesn't work when I do it dynamically later this week, but, I wanted to get through as many of the tests as I could first...

or do we still have random chrashes using the dynamic runtime library?
Especially if not: can you please upload current test results?

[LADSoft]

yeah i got the dynamic runtime library working before I left off with this. Will try to upload results sometime this weekend, I have to run them again...

meanwhile still having problems with the appveyor builds. Something up with the MSVC compiles. I suspect this is an entirely different issue from the random crashes I was seeing... I can't test with the version of the compiler we are currently using on appveyor so I'm going to bite the bullet and try the upgrade to the 2022 image again to see if that helps... guess if it works I will have to get serious about trying to get CLANG/MINGW64 to work on the new image...

[LADSoft]

here is the latest log - it was created with a version of cobc/libcob that was linked against the dynamic runtime library. A lot of the failures are still CR/LF related, it seemed like COBC_RUN has problems with initializing the COB_UNIX_LF, but when I did a cursory glance through one of the failing programs I couldn't see why it wouldn't work... it seemed like everything was linked properly against lscrtl.dll and the initialization routine should have been called albeit indirectly. I move on about then...

testsuite.log

[GitMensch]

Yes, there seem to be issues - but those look like "general", not only the COB_UNIX_LF setting.
if we take test 762 and leave CRLF out_

../../tests/run_misc.at:4173: COB_PRE_LOAD="prog"$PATHSEP"prog1" $COBCRUN_DIRECT ./main-prog
--- /dev/null	2023-09-08 12:44:29.000000000 -0400
+++ /c/autoconfigprojects/gnucobol-3.2_win/build_lscrtl/tests/testsuite.dir/at-groups/762/stderr	2023-09-08 12:44:29.662308600 -0400
@@ -0,0 +1 @@
+libcob: main-prog.cob:10: error: module 'ent3' not found
--- -	2023-09-08 12:44:29.769999400 -0400
+++ /c/autoconfigprojects/gnucobol-3.2_win/build_lscrtl/tests/testsuite.dir/at-groups/762/stdout	2023-09-08 12:44:29.709694900 -0400
@@ -1,4 +1,3 @@
 12abc
 11
-xxxxx

So either COB_PRE_LOAD does not work (then there's likely an issue with actual reading the environment) ... or PATHSEP is just not setup correctly, which is possibly the case.

Interesting is test 791 as this errors, likely within cobc (or within occ) - and we don't get a single output.
I highly suggest to inspect that, possibly running this test verbose to get a better clue where the issue happens make check TESTSUITEFLAGS="791" COBOL_FLAGS="-v"

Similar I'd highly suggest to inspect test 1124 - looks like either OCC or more likely the C runtime compiles offsets wrong:

-33331    22        333            444
-3333  3    22      111            000
+3333444  22        333               
+3333000    22      111               

Also interesting is test 808

../../tests/run_misc.at:10505: COB_DUMP_FILE=NONE \
$COBCRUN prog
--- -	2023-09-08 12:45:13.890638400 -0400
+++ /c/autoconfigprojects/gnucobol-3.2_win/build_lscrtl/tests/testsuite.dir/at-groups/808/stderr	2023-09-08 12:45:13.827941600 -0400
@@ -1,3 +1,4 @@
 libcob: cpyabrt:4: error: LINKAGE item 'TSPFL-RECORD' (accessed by 'CM-COMPANY') not passed by caller
 libcob: cpyabrt:4: warning: implicit CLOSE of FLATFILE ('RELFIX')
+libcob: cpyabrt:4: warning: issue during unlock (cob_file_close), errno: 6

We'd need to check if that happens with an MSVC version (it doesn't happen with GCC), this errno 6 is special.

[LADSoft]

I'll get back to this maybe next weekend, also maybe I'll take another look at what is going on with the CR/LF. I've been mired for several days in issues that came up because of my recent rewrites on another branch and am trying to get that all sorted out so we can have a release for #603 and maybe get that issue put to bed.

[LADSoft]

I kinda took a break last week and then worked on a compiler bug that was impeding the release and did some C23 work... I'll get to this soon...

[LADSoft]

i fixed the cr/lf issue, then the other three issues you flagged went away.

here is the current log (11 unexpected failures, 5 unexpected passes)
testsuite.log

[GitMensch]

What was the issue with the CR/LF?

Are the results with the static and shared runtime library similar now?

For the tests results - the following looks very strange, maybe there is some debug output in the runtime library?

776. run_misc.at:4974: testing CALL in from C with init missing / implicit ...
../../tests/run_misc.at:5011: $COMPILE -o caller caller.c callee.cob
../../tests/run_misc.at:5012: $COBCRUN_DIRECT ./caller
../../tests/run_misc.at:5016: $COMPILE -fimplicit-init -o caller caller.c callee.cob
../../tests/run_misc.at:5017: $COBCRUN_DIRECT ./caller
--- -	2023-09-17 21:42:05.186317900 -0400
+++ /c/autoconfigprojects/gnucobol-3.2_win/build_lscrtl/tests/testsuite.dir/at-groups/776/stdout	2023-09-17 21:42:05.118694400 -0400
@@ -1 +1,2 @@
+mgeinit
 OK

Other than that test 931 "EXTFH: changing record address" with its result "Segmentation fault" should definitely be inspected (maybe you can have a look?)
I'll recheck test 941 "Concatenated Files" with its "warning: issue during unlock (cob_file_close)" on a VC environment.

[LADSoft]

i thought I'd gotten all the debug printfs out of libcob\common.c... that is where +mgeinit came from. I've pulled it out again now so I guess that test will pass next time.

i had a lot of problems with segmentation faults yesterday, didn't realize there was still one left. I think I looked at this before but don't remember how I left it. Will check it out soon...

the problem with the CR/LF was a bug in the shared runtime library. It was allowing itself to be initialized multiple times and one of those times happened to be after the unix lf mode was set up in a lot of the cobcrun tests... it went back to windows lf every time it was reinitialized.

from memory, the shared runtime library actually ends up with significantly less failed tests with the libcob code as originally written. Even though I had made some changes to cobc/cobcrun to address various CR/LF issues with the non-shared library, there were still situations where what I did wasn't enough... using the shared library seems to address all the issues I was having with that both from cobc/cobcrun and from other sources. I think other than that we are probably getting comparable results.

I did rerun 'omake test' last night. Seems like it is still working fine...

[GitMensch]

So ... how does the configure line look like when using occil? And: what are the test results there?

[LADSoft]

hehehe you still wanna push the envelope? Well there are more problems with #603, works great if the compiler isn't compiled with itself, but once we compiled it with itself there were some problems with the debug info generation causing the link to fail. I'll work on that then get back to this later...

[LADSoft]

so i looked at #931, the problem seems to be in fileio.c/indexed_open on the following line:

	isfd = isopen ((void *)filename, omode | lmode | vmode);

i verified the filename is correct (I'm assuming it shouldn't have an extension when specified here even though the file pair on disk has extensions) and the flags come out as ISKEEPLOCK|ISVARLEN|ISINOUT

the open fails with isfd = -1.

This leaves the 'file' pointer in the cob_file struct as null, and when an indexed_read is done later on (apparently the open isn't checked) this field is dereferenced resulting in the segment violation.

So this seems like it is some issue with vbisam... if you concur we are done with this aspect of the tests.

[LADSoft]

im working now with occil... I see lots of problems ahead but for right now I just wanna get the configure script to compile and run a file. The file it tries to run uses fopen. This file compiles ok, but crashes when run. I tried extracting the code from the 'configure' script and compiling it but that works fine when run. So I'm wondering, is there some way to get the configure script to keep the files it is working with so I can try to diagnose this further?

[GitMensch]

I don't know of a way to keep those, but any failing test is documented in config.log, including the generated config.h and the command used to run the test. Note: cool that you give that a try!

[LADSoft]

so i got a little further after I realized that the way I called the confgure script was broken... now it is stalling because olib won't handle the object files from occil... there is going to be another similar issue when I try to link because right now occil has a built-in linker but the way this is going to work with autoconfig it will have to call the external one and give it a list of object files... but I am at least very impressed because the configure script seems to have correctly inferred the object file extension for this 😄

because of the scope of work (we need a librarian!) I will leave this for now and come back to it later.

[GitMensch]

Note: occil may be the librarian, too, that may be the easiest option

[LADSoft]

im not sure it would be... occil doesn't read object files, it just compiles then generates an executable based on the internal database of what was compiled. Or it can generate object files, which can then be passed to a more traditional linker. The weird part is the psuedo-link it can do on its own... Im thinking unless I want to import a lot of the traditional linker and librarian support into the occil backend, I would have to have an external librarian. And Im kinda torn on the idea of throwing so much functionality into the compiler backend; it probably has too much functionality already...

[GitMensch]

Ok, then a new librarian. While it isn't the wanted end result, maybe one can use olink in the meantime and pass extra libraries as needed, even if it would end in native executables (to get a bit further)?

[LADSoft]

lol... unfortunately occil doesn't generate object files that are compatible with olink... I'll get back to this soon, gonna work on c++ 17 for a while first though...