Add Crystal support #2732

HertzDevil · 2021-06-23T05:04:53Z

Resolves #673. Includes compiler versions from 1.0 down to 0.29.

The Monaco language is written from scratch and supports only basic highlighting; Crystal's syntax is borrowed from Ruby, but there are too many dissimilarities between the two and I'd prefer lack of e.g. indenting rules over incorrect ones. It is subject to future improvements.

Even an empty file produces 190k lines of assembly code, unless --prelude=empty is provided (which more or less excludes the entire runtime and all libraries). If this is unacceptable then arrangements would have to be made on the language side.

Related: compiler-explorer/infra#533

partouf

Thanks so much for doing this! Looks good

static/modes/crystal-mode.js

lib/compilers/crystal.js

partouf · 2021-06-23T07:47:19Z

Some issues that need to be resolved before this:
When I install one of the versions (0.29.0 for example) and try to run a simple example, I get these errors:

partouf@ceub20:/tmp$ /opt/compiler-explorer/crystal-0.29.0/bin/crystal build -o output.s --emit asm hello.cr
/usr/bin/ld: cannot find -lpcre (this usually means you need to install the development package for libpcre)
/usr/bin/ld: cannot find -levent (this usually means you need to install the development package for libevent)
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc "${@}" -o '/tmp/output.s'  -rdynamic  -lpcre /opt/compiler-explorer/crystal-0.29.0/bin/../lib/crystal/lib/libgc.a -lpthread /opt/compiler-explorer/crystal-0.29.0/share/crystal/src/ext/libcrystal.a -levent -lrt -ldl -L/opt/compiler-explorer/crystal-0.29.0/bin/../lib/crystal/lib -L/usr/lib -L/usr/local/lib`

When running 1.0.0 I get less errors, but errors nonetheless:

partouf@ceub20:/tmp$ /opt/compiler-explorer/crystal-1.0.0/bin/crystal build -o output.s --emit asm hello.cr
/usr/bin/ld: cannot find -lpcre (this usually means you need to install the development package for libpcre)
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc "${@}" -o /tmp/output.s  -rdynamic -L/opt/compiler-explorer/crystal-1.0.0/bin/../lib/crystal/lib -lpcre -lm -lgc -lpthread /opt/compiler-explorer/crystal-1.0.0/share/crystal/src/ext/libcrystal.a -levent -lrt -ldl`

It seems to me crystal requires a C compiler in the path, that should be possible with the default Ubuntu 20.04 gcc compiler (though not sure if that's what we want always) - but the main problem lies in the non-standard prerequisite libraries (through the OS package system).

So my questions would be:

Is there a list of these requirements per versions somewhere?
Can we make it less dependent on cc and -L/usr/lib -L/usr/local/lib? - although I'm guessing maybe 1.0.0 has a different library resolving mechanism

partouf · 2021-06-23T08:06:50Z

It seems there's this list
https://github.com/crystal-lang/crystal/wiki/All-required-libraries

It also seems to require an LLVM to be installed?

And the latest documentation seems to refer to some environment variables for the build command:
https://crystal-lang.org/reference/using_the_compiler/index.html#environment-variables

But I don't know if these are also used in the earlier versions.

partouf · 2021-06-23T08:20:31Z

Note that the biggest issue here would probably requiring library installs on the OS level, as we would need to produce new images for our instances. We can probably solve the CC and LLD requirements some other way.

cc @mattgodbolt

HertzDevil · 2021-06-23T09:13:30Z

cc can be overridden through the CC environment variable, and library search paths are prepended by CRYSTAL_LIBRARY_PATH. The required libraries have not been changed from that list you found for the Crystal versions added in this PR.

LLVM shouldn't be needed just for running Crystal. It definitely needs to be there when building Crystal from Crystal itself (something to consider for nightlies).

straight-shoota · 2021-06-23T09:53:56Z

It seems to me crystal requires a C compiler in the path

Crystal just requires a linker. It picks up what's configured as CC (and defaults to cc). We can use whichever fits for this purpose. The compiler also allows passing flags via --link-flags CLI option.

Is there a list of these requirements per versions somewhere?

So https://github.com/crystal-lang/crystal/wiki/All-required-libraries is a list of all libraries required for the entire standard library and compiler. If you include the XML module, you need libxml2 for example.
I assume it's probably best to have all stlib dependencies available. Not sure about the compiler dependencies (the difference is only LLVM, though).

It also seems to require an LLVM to be installed?

LLVM is linked statically into the compiler binary from the tarball. Unless building the compiler itself, there is no need to have LLVM installed.

And the latest documentation seems to refer to some environment variables for the build command:
crystal-lang.org/reference/using_the_compiler/index.html#environment-variables

But I don't know if these are also used in the earlier versions.

They should be available in earlier versions, but probably not all back in 0.29. I'm not sure if there's much point in going so far back, anyways. IMO it would be fine to focus on 1.0, at least for a first iteration.

HertzDevil · 2021-06-23T10:00:19Z

The one I mentioned, CRYSTAL_LIBRARY_PATH, seems to have existed since 0.28.0.

Perhaps we could build the libraries locally with https://github.com/compiler-explorer/infra/blob/main/update_compilers/install_libraries.sh and symlink to them from crystal-{name}/lib/crystal/lib? Theoretically we could even offer the same libraries for C/C++.

Also added an assembly parser that handles Crystal's quoted labels. Filtering everything reduces the number of lines to roughly 13k (the Crystal prelude's own included files are counted as library code).

HertzDevil · 2021-06-24T04:52:09Z

Will this work?

caspiano · 2021-06-26T01:46:58Z

static/modes/crystal-mode.js

+        ],
+
+        surroundingPairs: [
+            { open: '{', close: '}' },


Suggested change

{ open: '{', close: '}' },

{ open: '{', close: '}' },

{ open: '{%', close: '%}' },

Not sure I'm reading this correctly, but I think macro block identifiers are missing.

RubenRBS

Thank you so much for this!

RubenRBS

Forgot to actually send the review

etc/config/crystal.amazon.properties

partouf · 2021-06-27T11:44:12Z

Superweird asm emission, but c'est la vie.

The 1 thing I'm worried about is that the square function example is unused and is not emitted until you call the function.

partouf · 2021-06-29T08:02:39Z

Superweird asm emission, but c'est la vie.

The 1 thing I'm worried about is that the square function example is unused and is not emitted until you call the function.

Any thoughts about the last one? This is different behaviour compared to all the other languages and compilers.
There doesn't seem to be an extern/export kind of option (at least not what I can find in the language documentation)

Perhaps an option would be to get an input parameter from the command line and give that to the function, so that it's always used?

straight-shoota · 2021-06-29T08:55:15Z

Export function would be fun instead of def. But then it can only use primitive types as parameters. Int32 works, for example.

fun square(num : Int32) : Int32
  num * num
end

But I don't think that would be very helpful for general use cases.

The general approach to include a Crystal method in codegen is adding a call to it.

partouf · 2021-06-29T11:50:40Z

Export function would be fun instead of def. But then it can only use primitive types as parameters. Int32 works, for example.
fun square(num : Int32) : Int32
  num * num
end
But I don't think that would be very helpful for general use cases.

The general approach to include a Crystal method in codegen is adding a call to it.

Any idea on how to do that without specifying a specific number? I saw you can override the main() function, but doubt that's a recommended thing to do

straight-shoota · 2021-06-29T15:09:38Z

Sorry, I actually have no idea how compiler explorer works and what we're aiming for here. If the method is to be called without a baked-in value, the value needs to come from somewhere like a CLI arg, env or rand.

There shouldn't be a reason to override main, because everything in the top level scope is executed as main body.

partouf · 2021-06-29T15:30:55Z

Sorry, I actually have no idea how compiler explorer works and what we're aiming for here. If the method is to be called without a baked-in value, the value needs to come from somewhere like a CLI arg, env or rand.

There shouldn't be a reason to override main, because everything in the top level scope is executed as main body.

I didn't mean how to get the value from CE, that's already supported for cmdline args and stdin.

My question was about how to write such an example in Crystal that uses it.

partouf · 2021-07-02T16:38:39Z

Sorry, I actually have no idea how compiler explorer works and what we're aiming for here. If the method is to be called without a baked-in value, the value needs to come from somewhere like a CLI arg, env or rand.
There shouldn't be a reason to override main, because everything in the top level scope is executed as main body.

I didn't mean how to get the value from CE, that's already supported for cmdline args and stdin.

My question was about how to write such an example in Crystal that uses it.

@HertzDevil can you create such an example? (something that uses cmdline input and calls square with it)

…orer into crystal

mattgodbolt · 2021-07-06T13:05:02Z

Catching up here, agree with everything @partouf has said. The intention of the default example is to show a little simple assembly, conventionally the "square a number" routine. It's not meant to be a language primer; is there a reason the fun isn't OK? We can have named examples (accessible from the load/save button) that show off the language more, if that's what you want.

partouf · 2021-07-06T13:26:44Z

Catching up here, agree with everything @partouf has said. The intention of the default example is to show a little simple assembly, conventionally the "square a number" routine. It's not meant to be a language primer; is there a reason the fun isn't OK? We can have named examples (accessible from the load/save button) that show off the language more, if that's what you want.

The problem is that the square fun without a call isn't emitting Any assembly, so there would be nothing but overhead showing.
If it would emit something that resembled c=a*b, then you would be absolutely right, but it doesn't.

So I think this is the way to go.

I have tested the rest, so this is good to merge and go live.

straight-shoota · 2021-07-06T14:01:03Z

The intention of the default example is to show a little simple assembly,

In that case, maybe it would be an option to run the example without stdlib. That would keep the generated assembly to a minimum and more strictly represent the language itself instead of the stdlib. @HertzDevil WDYT?
It might be difficult to achieve because this requires a different compiler arg (ˋ--prelude=emptyˋ) than building any "normal" Crystal code which would usually include the stdlib.

HertzDevil · 2021-07-06T14:05:40Z

I think all funs will emit code even if they aren't "instantiated" like defs. I'll try to come up with an example a few hours later.

Is it possible to pre-populate the compiler args when the default snippet is loaded?

partouf · 2021-07-06T14:13:25Z

I think all funs will emit code even if they aren't "instantiated" like defs. I'll try to come up with an example a few hours later.

Ah, I think I missed the point of changing def to fun. I don't remember if I tested that.

Is it possible to pre-populate the compiler args when the default snippet is loaded?

Not afaik. We've talked about it in various circumstances, but it's just not worth the trouble imo.

straight-shoota · 2021-07-06T15:31:15Z

Yes, fun Methode are exported always. But with default compiler behaviour, there's still a lot of additional code for runtime initialization.

HertzDevil · 2021-07-06T19:39:43Z

An absolute minimal example for that would be:

# compile with --prelude=empty

fun square(num : Int32) : Int32
  num &* num
end

In this case the ASM output would just be:

square:
        imull   %edi, %edi
        movl    %edi, %eax
        retq

".L'dd40a2442'":
        .long   1
        .long   9
        .long   9
        .asciz  "dd40a2442"
        .zero   2

; ...

Does this look fine?

partouf · 2021-07-06T20:06:11Z

An absolute minimal example for that would be:

# compile with --prelude=empty

fun square(num : Int32) : Int32
  num &* num
end

In this case the ASM output would just be:

square:
        imull   %edi, %edi
        movl    %edi, %eax
        retq

".L'dd40a2442'":
        .long   1
        .long   9
        .long   9
        .asciz  "dd40a2442"
        .zero   2

; ...

Does this look fine?

absolutely, that's fine

* Add Crystal support * Fix copyright * Add ASM parser for Crystal * Add supportsLibraryCodeFilter * Update crystal-mode.js * use baseName * Force `square` call * Update default snippet Co-authored-by: Patrick Quist <partouf@gmail.com>

Add Crystal support

3c346f5

HertzDevil requested review from jaredwy and mattgodbolt as code owners June 23, 2021 05:04

github-actions bot added the ui label Jun 23, 2021

HertzDevil mentioned this pull request Jun 23, 2021

Add installers for Crystal compiler-explorer/infra#533

Merged

partouf requested changes Jun 23, 2021

View reviewed changes

static/modes/crystal-mode.js Outdated Show resolved Hide resolved

lib/compilers/crystal.js Outdated Show resolved Hide resolved

Fix copyright

3a0fb6c

HertzDevil added 2 commits June 23, 2021 18:03

Add ASM parser for Crystal

d682bca

Add supportsLibraryCodeFilter

a04557a

Update crystal-mode.js

8f6a53e

caspiano reviewed Jun 26, 2021

View reviewed changes

RubenRBS reviewed Jun 26, 2021

View reviewed changes

RubenRBS requested changes Jun 26, 2021

View reviewed changes

etc/config/crystal.amazon.properties Outdated Show resolved Hide resolved

use baseName

75c2617

Merge branch 'main' into crystal

bd9a716

Force square call

36f8fcf

Merge branch 'crystal' of https://github.com/HertzDevil/compiler-expl…

9fec5dd

…orer into crystal

Update default snippet

946cc9d

partouf approved these changes Jul 7, 2021

View reviewed changes

HertzDevil requested a review from RubenRBS July 7, 2021 09:06

partouf merged commit 94ecb09 into compiler-explorer:main Jul 7, 2021

HertzDevil deleted the crystal branch July 7, 2021 22:29

HertzDevil mentioned this pull request Jul 12, 2021

Override CC environment variable in Crystal propeties #2776

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Crystal support #2732

Add Crystal support #2732

HertzDevil commented Jun 23, 2021 •

edited

partouf left a comment

partouf commented Jun 23, 2021 •

edited

partouf commented Jun 23, 2021

partouf commented Jun 23, 2021 •

edited

HertzDevil commented Jun 23, 2021

straight-shoota commented Jun 23, 2021

HertzDevil commented Jun 23, 2021 •

edited

HertzDevil commented Jun 24, 2021

caspiano Jun 26, 2021

RubenRBS left a comment

RubenRBS left a comment

partouf commented Jun 27, 2021

partouf commented Jun 29, 2021

straight-shoota commented Jun 29, 2021

partouf commented Jun 29, 2021

straight-shoota commented Jun 29, 2021 •

edited

partouf commented Jun 29, 2021

partouf commented Jul 2, 2021

mattgodbolt commented Jul 6, 2021

partouf commented Jul 6, 2021

straight-shoota commented Jul 6, 2021

HertzDevil commented Jul 6, 2021

partouf commented Jul 6, 2021

straight-shoota commented Jul 6, 2021

HertzDevil commented Jul 6, 2021 •

edited

partouf commented Jul 6, 2021

	{ open: '{', close: '}' },
	{ open: '{', close: '}' },
	{ open: '{%', close: '%}' },

Add Crystal support #2732

Add Crystal support #2732

Conversation

HertzDevil commented Jun 23, 2021 • edited

partouf left a comment

Choose a reason for hiding this comment

partouf commented Jun 23, 2021 • edited

partouf commented Jun 23, 2021

partouf commented Jun 23, 2021 • edited

HertzDevil commented Jun 23, 2021

straight-shoota commented Jun 23, 2021

HertzDevil commented Jun 23, 2021 • edited

HertzDevil commented Jun 24, 2021

caspiano Jun 26, 2021

Choose a reason for hiding this comment

RubenRBS left a comment

Choose a reason for hiding this comment

RubenRBS left a comment

Choose a reason for hiding this comment

partouf commented Jun 27, 2021

partouf commented Jun 29, 2021

straight-shoota commented Jun 29, 2021

partouf commented Jun 29, 2021

straight-shoota commented Jun 29, 2021 • edited

partouf commented Jun 29, 2021

partouf commented Jul 2, 2021

mattgodbolt commented Jul 6, 2021

partouf commented Jul 6, 2021

straight-shoota commented Jul 6, 2021

HertzDevil commented Jul 6, 2021

partouf commented Jul 6, 2021

straight-shoota commented Jul 6, 2021

HertzDevil commented Jul 6, 2021 • edited

partouf commented Jul 6, 2021

HertzDevil commented Jun 23, 2021 •

edited

partouf commented Jun 23, 2021 •

edited

partouf commented Jun 23, 2021 •

edited

HertzDevil commented Jun 23, 2021 •

edited

straight-shoota commented Jun 29, 2021 •

edited

HertzDevil commented Jul 6, 2021 •

edited