First Class Functions and Type System

[LouisJenkinsCS]

One thing I'm going to concede here is that cclock can't be improved at this point in time. The object class is the only way to do this, and any other way produces annoying and frustrating compiler errors (internal) or syntax errors which lead to paradoxes (like Chicken-And-The-Egg, except with types) where if you don't know the type at initialization time, or if the type can change, it just won't work. In this way, only class instances can work, which require wrapper objects. I spent far too long on attempting to improve it, I'll try later.

[LouisJenkinsCS]

I managed to 'fix' (read: work around) the issues with function pointers by inserting the logic inline with the code for CCQueue. This means that the operations and the cclock in general is now overly specific and hard to test by itself, as it is impossible to test in a modular fashion (also Chapel 1.16 master broke the queue, causing a compiler error ALI0073, of which I have no idea where that is (I kind of wish it just told you the file name).

In reality, I'm spent an entire day working on getting back to where I was before because of some technical difficulties in the compiler, and it's extremely frustrating as evaluations are just around the corner :(

[LouisJenkinsCS]

@e-kayrakli

[LouisJenkinsCS]

Can confirm: While compiling it by itself does not yield a crash, compiling it with other files causes the following internal error:

internal error: DEA0464 chpl Version 1.16.0 pre-release (3eaee12)

So any first class functions cause internal compiler issues. So unfortunate though.

[e-kayrakli]

OK, relax :)

Compile with --devel, it'll show you the assert that triggers in compiler. How do I reproduce this? Tell me more and I can play around/create an issue if necessary.

[LouisJenkinsCS]

Wow... so --devel flag is what I needed all along! In trunk, the make file (along with uncommenting all of cclock.chpl) will produce that error, but with the --devel flag I can actually see the issue (deadCodeElimination.cpp:464)! Wow, I wonder if I should just have it on by default for now on, would've saved me a lot of time.

[e-kayrakli]

Alternatively, you can set environment variable CHPL_DEVELOPER=1...

Sometimes it is a bit confusing though, because when there is a user error, it will generally show the exact halt location in internal modules, rather than the actual place in the user code.

Also, are you able to workaround your problem now? Or should I still take a look at it? (It may have to wait until tomorrow)

[LouisJenkinsCS]

You've done more than enough Engin, I should be able to diagnose and resolve any other issues I have for now! Greatly appreciate it!

[e-kayrakli]

No problem! It'd still be nice to see what was to problem and create a bug report if necessary, though. If you can get to the bottom of the problem (a minimal snippet, for example) let me know..

[LouisJenkinsCS]

@e-kayrakli

I see it! The issue! It seems that... it wasn't... even my code? When I have included BigInteger like such: use BigInteger and if it is in code that is never used (hence Dead Code Elimination), it actually crashed.

Create a simple file with just use BigInteger, and then create another file with a main function where execution begins, with a simple writeln.

For reference, I have it here as a gist.

This leaves me at such a loss, because I'm not sure if the issue was ever present in first class functions... meaning I wasted so much time...

Output:

LouisJenkinsCS@MSI:/mnt/c/Github/Distributed_Queue$ chpl --fast tmp/test.chpl tmp/test2.chpl
internal error: DEA0464 chpl Version 1.16.0 pre-release (3eaee12)

Internal errors indicate a bug in the Chapel compiler ("It's us, not you"),
and we're sorry for the hassle.  We would appreciate your reporting this bug --
please see http://chapel.cray.com/bugs.html for instructions.  In the meantime,
the filename + line number above may be useful in working around the issue.

Output w/ Developer Debug:

LouisJenkinsCS@MSI:/mnt/c/Github/Distributed_Queue$ chpl --fast --devel tmp/test.chpl tmp/test2.chpl
internal error: Expected initFn for module 'BigInteger', but was null [deadCodeElimination.cpp:464]

[e-kayrakli]

use ing DeadCodeEliminationBug from the enabler seems to get rid of the bug. Haven't inspected the compiler code, but this may be worthy of an issue. Will take care of it, thanks.

[LouisJenkinsCS]

Do you think this bug is specific to the BigInteger module? I'm assuming most people don't include files in compilation that isn't used (my makefile actually grabs any .chpl files and compiles them together, then tells it what is the --main-module, which may be an overlooked issue)

[e-kayrakli]

Yes. There are interesting pragmas in BigInteger, that by looking at their names control something that may be triggering this problem.

I am pretty sure this is a bug, but a super corner case bug.

[LouisJenkinsCS]

There's another issue with first class functions, but at this point would you prefer I make an actual issue of it on Chapel's issue tracker?

I tried making a minimal, but reasonable/practical application of it in the example: here. If you want to see if its worthwhile of being a bug, that's fine too.

[e-kayrakli]

Firstly, minimal working example(MWE) doesn't need to be practical at all, at least for bug reports. Here is a true MWE of the code you have:

proc foo() {
  var a: [{1..10}] int;
  return lambda(n:int) { return a[n]; };
}
writeln(foo()(1));

I think any internal bug error that you get without --devel is important to record for Chapel's development. When it comes to this one the issue is probably missing compiler error message. The problem occurs because you are using an array in the nested lambda and returning it. Now since you are using the outer array a, it needs to be passed by ref to the inner lambda. Therefore, compiler writes the lambda with an additional formal argument to pass that array by ref. However, you are calling that returned lambda without passing anything to that by-ref formal. This case is never checked where lambda signatures are created, and then compiler blows up at a much later point.

In sum, I think this is user error where compiler doesn't do a good job detecting it. You cannot use outer-local variables in a nested lambda and return it, because those variables will be reclaimed once the outer function returns the inner lambda.

From a more philosophical standpoint, lambdas/function pointers shouldn't be used for such complicated cases in a language like Chapel, because things get elusive and error-prone quite easily.

[LouisJenkinsCS]

I guess really I have to actually give up the usage of any lambdas (especially for my use cases). I'm amazed that you could find the root cause so quickly. I'll add any other bugs I find to issue #9