Add support for Swift #118

Closed
nafehshoaib opened this Issue Dec 14, 2015 · 43 comments

Projects

None yet

5 participants

@nafehshoaib

Since Swift is now open source, it would be great if we could use it on DMOJ! Here are the links for Swift:

developer.apple.com/swift/
www.swift.org/

@Xyene
Member
Xyene commented Dec 14, 2015

It is? I've been waiting for this for so long! I'll add it shortly.

@Xyene
Member
Xyene commented Dec 16, 2015

Support for Swift has been added on Herculaneum. However, Apple only provides binaries for 64-bit Ubuntu 14.04/15.10, but not 32-bit Debian (which Herculaneum is using).

I built Swift from the source of a fork of Swift with a hack for 32-bit compilation, https://github.com/nwellnhof/swift: it is currently 12 days behind the master repository, and is unlikely to be updated. It should, however, be good enough for most use cases until the time comes that Apple officially starts supporting 32-bit Swift. The exact build used on Herculaneum can be downloaded from binaries.dmoj.ca.

Note that this build of Swift does fail a couple of unit tests - more details can be found on the swift-dev mailing list.

@Xyene Xyene added a commit that referenced this issue Dec 16, 2015
@Xyene Xyene Add basic Swift executor; #118 3b918ff
@Xyene
Member
Xyene commented Dec 16, 2015

Looks like Foundation isn't compiled/available, so there's currently no way to input data.

import Foundation

print(NSString(data: NSFileHandle.fileHandleWithStandardInput().availableData, encoding:NSUTF8StringEncoding))

CE:

/tmp/tmpqkNKm4/echo.swift:1:8: error: no such module 'Foundation'
import Foundation
       ^
@WallE256
Member

I just installed Swift on my ubuntu server, and Foundation is available.

On Wed, Dec 16, 2015 at 6:01 PM, Tudor Brindus notifications@github.com
wrote:

Looks like Foundation isn't compiled/available, so there's currently no
way to input data.

import Foundation
print(NSString(data: NSFileHandle.fileHandleWithStandardInput().availableData, encoding:NSUTF8StringEncoding))

CE:

/tmp/tmpqkNKm4/echo.swift:1:8: error: no such module 'Foundation'
import Foundation
^


Reply to this email directly or view it on GitHub
#118 (comment).

@Xyene
Member
Xyene commented Dec 16, 2015

Awesome! I assume you used a 64-bit official binary, right? In that case, good to know it works on 64-bit.

@nafehshoaib

You don't need Foundation to input data in the console anymore. You can use the new readline() function introduced in Swift 2.0. You just need the Swift standard libraries.

var input = readline()

https://developer.apple.com/library/ios/documentation/Swift/Reference/Swift_StandardLibrary_Functions/index.html#//apple_ref/swift/func/s:FSs8readLineFT12stripNewlineSb_GSqSS_

@Xyene Xyene added a commit that referenced this issue Dec 16, 2015
@Xyene Xyene Use readLine for echo test; #118 b05318f
@Xyene
Member
Xyene commented Dec 16, 2015

That seems to work well, thanks. On the other hand, is Foundation good/necessary to have available? (Disclaimer: I know next to nothing about Swift)

@brunofalmeida

You need the Foundation framework for almost all problems. The standard readLine() function works for inputting one line, but you need the NSString class in Foundation for its string input/manipulation methods (separating by spaces, searching, etc.).

@brunofalmeida

On another note, the grading page after submitting in Swift says it's processing the submission until you refresh the page (it only gives your results after you refresh). This isn't very important, but if you can fix it then great.

@brunofalmeida

I believe the Foundation framework needs to be packaged separately and referenced by the Swift compiler/runtime.

https://github.com/apple/swift-corelibs-foundation

@nafehshoaib

Good Point! Completely forgot about that! Is there a 32 bit port?

@nafehshoaib

There is a way though to separate strings using spaces without Foundation.

var input = readLine()
var xA = input?.characters.split { $0 == " " }.map(String.init)
@Xyene
Member
Xyene commented Dec 17, 2015

The grading page has been fixed by @quantum5 (the event server was down; not a code issue).

Foundation is proving to be difficult to get working. I guess when Apple said they supported Ubuntu 14.04+, they really meant it. Swift comes with the option to build swift-corelibs-foundation through utils/build-script --foundation (details here). Supposedly, since the main Swift is already set to build for 32-bit, it should might work.

Here comes in the fun part: whenever swift-corelibs-foundation is set to be compiled, our wonderful hosting provider SIGKILLs the clang that's compiling (perhaps because of the constant 2+ system load average)... I'm resorting to repeatedly invoking the build script whenever it exits - since ninja is incremental, it will eventually finish building, SIGKILL or not. After 3 hours of running, it has 414 source files left to build out of ~900 (and I really hope the resulting Foundation is usable and not in some way corrupted by the numerous interruptions during the build process).

@nafehshoaib

I'm pretty sure the main Swift project (this) doesn't support 32-bit.

@nafehshoaib

The problem with that is that it only contains the headers for Foundation, not Foundation itself. So, what I think it will try to do is build the 64-bit version of Foundation and try to apply it to the 32-bit version of Swift. What we need to do is find a 32-bit port of Foundation's Project or something.

@Xyene
Member
Xyene commented Dec 17, 2015

I've also modified the files for swift-corelibs-foundation to target i386 platforms by batch-replacing all occurrences x86_64 with i386 (which is really almost all that was needed to get Swift to compile). What will result will most certainly be a 32-bit binary - the jury is still out on whether it will actually work or not.

@nafehshoaib

Oh ok! Let's see if it works!

@quantum5
Member

I built swift with Foundation. And it's done, however...

import Foundation

print(NSString(data: NSFileHandle.fileHandleWithStandardInput().availableData, encoding:NSUTF8StringEncoding))
fatal error: availableData is not yet implemented: file Foundation/NSObjCRuntime.swift, line 80

And I am not turned on by the idea of a Mac judge.

@Xyene
Member
Xyene commented Dec 18, 2015

Relevant:

https://github.com/apple/swift-corelibs-foundation/blob/5f474bb948665676c57cb5d2ff8fe66db40eb985/Foundation/NSFileHandle.swift#L24

Still, this is about as good as we're going to get (these features aren't implemented in the official builds yet, either). https://github.com/apple/swift-corelibs-foundation/blob/master/Docs/Status.md may be used to consult which features are implemented for Linux.

Also note that these unimplemented errors occur at runtime, rather than compile-time (so it may lead to some confused users wondering why their otherwise correct code gets RTE).

@brunofalmeida

Well of course that's up to you whether you want to keep Swift in the language set until Foundation is fully supported on Linux. The Foundation framework is designed to be a platform-independent set of core functionality, although it's not fully implemented yet (as Tudor said; you can check the second link in his comment to see the development progress of specific components). I don't know how quickly Foundation is being developed for cross-platform use though.

Another idea: would the judge be able to scan submissions and throw a compile-time error if a program tries to import functionality that isn't available?

@Xyene
Member
Xyene commented Dec 18, 2015

swift-corelibs-foundation is under pretty rapid development, so I trust that any major issues will soon be resolved. "All" that would be required is a recompile (oh god).

Scanning submissions could in theory be implemented (by getting the AST of a submission through the -dump-ast flag and analyzing it), but I reckon it would be more trouble than it's worth. AST for reference:

(source_file
  (import_decl 'Foundation')
  (top_level_code_decl
    (brace_stmt
      (call_expr type='()' location=test.swift:3:1 range=[test.swift:3:1 - line:3:23] nothrow
        (declref_expr type='(Any..., separator: String, terminator: String) -> ()' location=test.swift:3:1 range=[test.swift:3:1 - line:3:1] decl=Swift.(file).print(_:separator:terminator:) specialized=no)
        (tuple_shuffle_expr implicit type='(Any..., separator: String, terminator: String)' location=test.swift:3:8 range=[test.swift:3:7 - line:3:23] sourceIsScalar elements=[-2, -1, -1] variadic_sources=[0]
          (paren_expr type='Any' location=test.swift:3:8 range=[test.swift:3:7 - line:3:23]
            (erasure_expr implicit type='Any' location=test.swift:3:8 range=[test.swift:3:8 - line:3:8]
              (call_expr implicit type='String' location=test.swift:3:8 range=[test.swift:3:8 - line:3:8] nothrow
                (constructor_ref_call_expr implicit type='(_builtinStringLiteral: RawPointer, byteSize: Word, isASCII: Int1) -> String' location=test.swift:3:8 range=[test.swift:3:8 - line:3:8] nothrow
                  (declref_expr implicit type='String.Type -> (_builtinStringLiteral: RawPointer, byteSize: Word, isASCII: Int1) -> String' location=test.swift:3:8 range=[test.swift:3:8 - line:3:8] decl=Swift.(file).String.init(_builtinStringLiteral:byteSize:isASCII:) specialized=no)
                  (type_expr implicit type='String.Type' location=test.swift:3:8 range=[test.swift:3:8 - line:3:8] typerepr='String'))
                (string_literal_expr type='(_builtinStringLiteral: Builtin.RawPointer, byteSize: Builtin.Word, isASCII: Builtin.Int1)' location=test.swift:3:8 range=[test.swift:3:8 - line:3:8] encoding=utf8 value="Hello, World!")))))))))

Anyways, Swift support looks like it's working as well as can be expected for something that has only a month or so of work under its belt. Barring any errors due to the hacky 32-bit compilation, it should be good for use until a more comprehensive implementation is completed.

@quantum5
Member

An AST says nothing about runtime. To accurately predict whether a runtime error would be produced requires actually simulate the execution, while I suspect would eventually force us to solve the halting problem.

@Xyene
Member
Xyene commented Dec 18, 2015

You don't actually need to predict whether a runtime error will occur, rather whether it has a chance of occurring due to unimplemented API. It's theoretically feasible to have a blacklist of functions (e.g., availableData) that are unimplemented, and return a compiler message warning users if they are referenced. Once again though, probably not worth it given the fast rate at which Foundation is being developed.

@Xyene
Member
Xyene commented Dec 18, 2015

Another point: having tested nothing but a simple hello world application, it is very possible that the sandbox is too strict and kills legitimate submissions.

@Xyene Xyene referenced this issue in DMOJ/site Dec 18, 2015
Closed

Swift syntax highlighting #353

@brunofalmeida

I was thinking it could just check for 'import Foundation', so would the judge be able to generate a warning for users to understand that the runtime error is related to Foundation?

@brunofalmeida

That is, if they get a runtime error with no other description, the compilation warning would indicate that it's likely from unimplemented Foundation classes.

@WallE256
Member

Asaik, the executor uses swiftc which throws CE if the class is not available.

Sent from my iPhone

On 18 Dec 2015, at 15:46, Bruno Almeida notifications@github.com wrote:

That is, if they get a runtime error with no other description, the compilation warning would indicate that it's likely from unimplemented Foundation classes.


Reply to this email directly or view it on GitHub.

@nafehshoaib

How about adding a commented block of text on top of their code when typing in DMOJ in Swift telling them that a lot of the classes in Foundation are unimplemented as of yet and to refer to a doc that lists them.

@Xyene
Member
Xyene commented Dec 18, 2015

Added a compile message warning of unimplemented API: https://dmoj.ca/submission/154617

It's not quite as nice as a block of code in the editor, but at least it doesn't require large changes to the site code.

@brunofalmeida

Yeah that seems to be the best thing to do for now, so that code still compiles but the user is aware that some code may not work. As more of it gets implemented, the Foundation framework should be updated periodically on the judge. Thanks for your efforts.

@Xyene Xyene added a commit that referenced this issue Dec 23, 2015
@Xyene Xyene readLine()!, not readLine(); #118 8cd4170
@Xyene
Member
Xyene commented Dec 23, 2015

We have a new judge, Laputa, which runs 64-bit Debian 8. I've installed Swift there and it appears to work.

@nafehshoaib

OHH YESSS! I'll give it a go!

@brunofalmeida

If you have a working 64-bit version of Swift on Laputa, is it still advisable to have the "hacked" 32-bit version running on Herculaneum? I didn't have any problems with the 32-bit version before, but since 32-bit isn't officially supported I'm wondering if anyone found problems with it.

@quantum5
Member

@Xyene and I are unlikely to remove the hacked 32-bit Swift very soon because we find it hard to remove such a piece of hard work.

@nafehshoaib

I can't find a problem that uses Laputa. Throughout the week I did some problems with 32-bit Swift on Herculaneum and Foundation was basically non-existent. Kept giving me errors for everything, even methods that are already implemented in the new Foundation.

@Xyene
Member
Xyene commented Dec 25, 2015

Almost all problems grade on Laputa as well. If you get an error about something not being implemented, it's likely because it's not (or because the official build has not been updated). You can run the following Python code to determine which judge a problem is grading on:

import socket
print socket.gethostname()

If you're checking a problem, you should Ctrl+spam click the submit button, so that multiple submissions will get queued at the same time. A hack, sure, but it should prove that the problems are grading on Laputa, with the latest Swift build.

@nafehshoaib

Where do I type this in? Also, on the problem submission page, it shows Swift to be using the i386 build.

@Xyene
Member
Xyene commented Dec 25, 2015

You can just type it in the submit box for any problem. The language dropdown often is inaccurate, since it is a global setting for something that is local to the judges. Indeed, Herculaneum continues to be using the i386 build, while Laputa is running

Swift version 2.2-dev (LLVM 3ebdbb2c7e, Clang f66c5bb67b, Swift 0ddf238ad7)
Target: x86_64-unknown-linux-gnu

even though the dropdown says i386. This is the case for many languages, where the actual runtime may be newer than the listed info (we update the judges regularly, the runtime info... not so regularly).

@Xyene
Member
Xyene commented Dec 25, 2015

@nafehshoaib I took a quick look through your submissions, and came across your solution to Bruno and Trig. It gets an IR with

gfsspc1p1: error while loading shared libraries: libdl.so.2: failed to map segmy
Exited with error: 127

In other words, it runs out of memory (MLE, but Swift reports as IR). I increased the memory to 64M, and your solution passes fine. Could you please link all the solutions you think are being graded incorrectly? That would help a lot in finding any possible bugs with the Swift executor.

@brunofalmeida

Is the hacked 32-bit version still a couple of weeks old or is the person who made it continually updating the build?

@Xyene
Member
Xyene commented Dec 26, 2015

The Swift version is a couple weeks old, but the Foundation build is recent (they're built separately).

@Xyene
Member
Xyene commented Jun 5, 2016

The Swift executor doesn't seem to have any problems after running it for a few months. The underlying runtime is a bit shaky, but otherwise this executor is good to go.

@Xyene Xyene closed this Jun 5, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment