Set up AppVeyor & NuGet packages

[qwertie]

I think we should have NuGet packages for each component: Loyc.Essentials, Loyc.Collections, Loyc.Syntax, Loyc.Utilities, Loyc.Ecs, LeMP and LLLPG. But right now there is only LoycCore (see Core/LoycCore.nuspec) and I've been too distracted to set up new packages. Hmm, I wonder if NuGet has a way of letting multiple people (U&I) have permission to update a given package. Anyway, since I have to update NuGet packages manually, I do it less often than I should.

I saw AppVeyor once and it looks cool... I am interested in having independent automated builds since it's too easy to push something that only works on my machine, or that has the .NET 3.5 build broken (which I guess I'll remove soon anyway... but after all other spring cleaning is done). But I always figured I'd have to give it some special thought because ideally you'd want to not just build EC#/LeMP but also make sure that it compiles itself correctly. It's a horribly manual process right now:

1. I build Release .NET 4 and close Visual Studio
2. I run the batch file that rebuilds the VS extensions and offers to reinstall them
3. I reopen the solution and re-run LeMP on a bunch of source files
4. I use Git Extensions to check the diffs and make sure any changes are as expected (usually only the header comment changes to reflect the new version number)
5. If I have lots of presence of mind, I make sure it still builds OK
6. I push the new binaries

But just doing 1 & 2 in CI would be a nice start.

So @jonathanvdc, help setting up AppVeyor and NuGet packages would be appreciated. Do you know how to set up AppVeyor such that it can auto-increment the build number when changes are pushed? I saw a video where a guy had set up an auto-increment feature, but I have no idea how it works on a technical level.

Another thing we should do is set up a Loyc organization on GH to hold a repository of finished features, with our partly done features under our usernames. Except I don't know what to call the organization since some zero-commit jagoff took the name "Loyc".

[jonathanvdc]

I've forked ecsharp/master, and I got started on setting up AppVeyor. Right now, appveyor.yml just builds Loyc, and creates a NuGet package for LoycCore. Baby steps.

If you could provide me with a list of LeMP/LLLPG command-line instructions to run, then I'll add those to the appveyor.yml file.

Oh, I also can't get some projects to build because they have a rem pre-build command. I'm a Linux user, so xbuild crashes when it encounters rem. Hence, Loyc fails to build. I think that's kind of a shame. Mind if I eighty-six those rem commands?

AppVeyor increments the build number by default, and puts that in an environment variable. I'm using that right now in the appveyor.yml to append a suffix to the LoycCore *.nupkg.

Also, I don't think we really need a shared NuGet account, because I don't expect that any manual intervention will be necessary - or desirable, for that matter - once NuGet packages are pushed automatically.

[qwertie]

You mentioned a pre-build command before and I didn't know what you were talking about ... so I searched. Looks like the only REM is in Loyc.Syntax.csproj. It's a bootstrapping command I used three or four years ago. By all means, please remove it.

[qwertie]

Btw did you realize rem means "remark"? It's how you comment something out in DOS.

It's a matter of control, I want administrative control over the Loyc & EC# packages, if necessary. After logging into NuGet, click your username, then "Manage my packages", then click a package, and then "Manage Owners", then add "qwertie".

Hmm, an enviroment variable? How do we get that into the [assembly:AssemblyVersion]? We could add a macro that reads an environment variable into a string, and use it like

[assembly: AssemblyVersion    ("1.7.6."+getEnvVar(APPVEYOR_BUILD_NUMBER))]
[assembly: AssemblyFileVersion("1.7.6."+getEnvVar(APPVEYOR_BUILD_NUMBER))]

Or simply use whatever technique you're already using to insert the build number in the nupkg.

[jonathanvdc]

Btw did you realize rem means "remark"? It's how you comment something out in DOS.

Yeah, I know. But bash is completely unaware of that, so xbuild thinks it's an actual command, logs an error and stops.

Or simply use whatever technique you're already using to insert the build number in the nupkg.

I use nuget pack -Version %PKG_VERSION% Core\LoycCore.nuspec where PKG_VERSION is:

• SEMVER if a tag has been pushed
• %SEMVER%-ci%APPVEYOR_BUILD_NUMBER% otherwise

where SEMVER = 1.7.6 right now.

I've also configured AppVeyor to patch the AssemblyVersion.cs file with version 1.7.6.{build}.

It's a matter of control, I want administrative control over the Loyc & EC# packages, if necessary. After logging into NuGet, click your username, then "Manage my packages", then click a package, and then "Manage Owners", then add "qwertie".

Actually, I'd feel much more comfortable if you handled publishing the Loyc packages with your account. You are the author of the Loyc libraries, after all. I'll be sure to add you as an administrator if any part of ecsc ever becomes a package, though.

To that end, I created a commented-out deploy section in the appveyor.yml file that uploads packages to NuGet if a tag has been pushed. All it takes is for you to insert an encrypted version of your NuGet API key, and push a tag whenever you feel like publishing a NuGet package. If and when the CI branch gets merged with master, that is.

Speaking of which, when would you like me to send a PR? Should I get LeMP to process the EC# source code first, or would you rather do that yourself?

By all means, please remove it.

Done. With a few additional tweaks, I also managed to get Travis CI to compile Loyc as well.

[qwertie]

What's the dif between the web site you're using - Travis CI - and AppVeyor?

SEMVER? Fairly obviously I haven't been using semantic versioning, otherwise I'd be at version 25.2 or something like that. If we were to use semantic versioning, it would probably be better to version each component separately (although appveyor.yml it looks like the version is global - why is the version number stated twice?). And I feel like certain public classes ought to be excluded from semantic versioning - if an API is unstable or unfinished but usually not needed by external code, it would be nice not to have to increment the version with every change. Alternately these APIs could be internal rather than public (with InternalsVisibleTo). I know I "should" use semantic versioning, it's just that I'm used to the old ways of thinking, where incrementing the major version implies some major upgrade or dramatic API changes.

btw - the yml file mentions VLists\Test project... that's old and ripe to be deleted too.

Hmm, based on what's in the yml, I take it take it AppVeyor has some kind of assembly rewriter that updates the [assembly:AssemblyVersion] et al.

You need not automate the "horrible manual process" yet, but it would be nice to build the Visual Studio extensions ... hmm, is it even possible for AppVeyor to build a Visual Studio extension? Seems so - perhaps it'll Just Work. But the way it's set up in my repo doesn't seem ideal - currently I use a batch file to copy from Bin\Release.NET4 over to Lib\LeMP, and then the VS extension takes the binaries in Lib\LeMP as references. But you could send a PR even without building the VS extension.

I'm still not in a big hurry to integrate everything 'cause I still have to worry about updating EC# to support C# 7 syntax.

[jonathanvdc]

Travis CI builds projects on Ubuntu instead of Windows, so it uses the Mono/mcs/xbuild toolchain instead of .NET/csc/msbuild. In my experience, AppVeyor is better at pushing packages to NuGet, so I haven't instructed Travis to do that. Travis is useful if you want to avoid alienating developers who use non-Windows machines, though. It's also really easy to set up.

SEMVER is just a variable name, by the way. I don't use semantic versioning either for Flame, because it's a constantly evolving project. That's not entirely unprecedented: LLVM is known for regularly breaking the public API, as well.

I think that AppVeyor actually modifies the C# AssemblyVersion.cs source file (because I pointed it at that file).

I'm not sure if I'm the right person to get the Visual Studio extensions to compile. I don't even have a local machine to test them on, so I doubt that I can effectively coerce AppVeyor to build them.

On an unrelated note: I see that you're including binaries directly in the EC# repository at the moment. Have you considered exclusively using GitHub releases to store your LeMP/LLLPG binaries instead? That's the way I configured Flame. To build Flame in a CI environment, I just curl a dsc release from my releases page, and then instruct it to compile itself. This doesn't have to be painful for users who want to git clone ecsharp for themselves: a script can automate the process. I'm convinced that this approach has two advantages:

• The binaries that can be downloaded from the releases page should always be stable, and represent the minimal version of LeMP (dsc in my case) you need to bootstrap EC# (Flame in my case).
• Your git history will thank you.

Anyway, I'll send you a PR right now, so you get started on setting up Travis CI and AppVeyor CI for this repo. I can always send you another PR later that makes the CI jump through even more hoops.

[qwertie]

Well, you know, since LeMP&LLLPG are dependencies of the project, I thought it made sense to include them in source control, so that it's easy to set up after cloning, and if one goes to any point in history, one gets a compatible version of the binaries. But I guess this isn't very sustainable - the .git folder is 61MB and I don't know how much of that is binaries, but I don't want it to keep getting bigger.

I guess instead we could take a dependency on our own NuGet package. curl (which I've never used) is not part of Windows but it seems to be bundled with Git, so that's a reasonable option too.

[jonathanvdc]

Oh, sure. Downloading NuGet packages for LeMP/LLLPG during the CI process would work just fine as well.

I was thinking about running Loyc's tests during the CI process. The tests need to output a non-zero exit code for Travis/AppVeyor to consider them as failed, so I've changed the return type of RunTests.Run from void to bool, and modified the testing harness to accept command-line arguments, too. Are you okay with that?

I only managed to get Loyc.Essentials.dll's unit tests and the LeMP article examples to successfully run to completion. The other unit tests seemed to encounter a small handful of bugs, so I didn't include them in the CI process. Perhaps it'd make sense to move the failing tests elsewhere, so Travis and AppVeyor can run all unit tests that don't fail right now.

By the way, could you spare some time to review my pull request? If so, then I'd love to hear your feedback on the changes I made.

[qwertie]

For tests that are expected to stay failing for a long time, I set the Fails property (e.g. [Test(Fails = "I haven't thought of a solution to this problem")]). Let's simply not count those failures as "real" failures for CI purposes. And for generality, I think RunTests should return int (number of unexpected failures) rather than bool.

[qwertie]

BTW, LLLPG has a class of test failures I don't know how to fix. I think the problem is that the hashcode of a Symbol can vary between runs ... for some reason. As a consequence, the output of the parser generator can be nondeterministic, sometimes producing if (a) { if (b) {...} ... and other times producing equivalent but different code like if (b) { if (a) {...} .... I'm not sure what to do about the problem; one possibility is, I could amend the testing code to accept multiple expected outputs, but a better solution would be to squash the nondeterminism, but first I'd have to understand it...

[jonathanvdc]

All right, I've taken those attributes into account, and RunTests now returns int. Unfortunately, test suites 3, 4, 5, and 6 still fail on my machine, so I haven't instructed the CI engines to run them yet.

Eliminating nondeterminism sounds like a good idea in general. Perhaps sorting Symbol instances lexicographically with an ordered set would work?

[qwertie]

Whew. I've been fixing bugs for the last several days (some were hard to fix so it took awhile); I didn't fix the nondeterminism in LLLPG, but otherwise no tests should fail unexpectedly.

[jonathanvdc]

That's great news. I see that you've released a new version of LeMP, as well. While I was reading the release notes, I noticed that there's a macro that "backports" the C# 6.0 null dot operator.

Which was a pleasant surprise, because, theoretically, ecsc could just insert a using LeMP.CSharp6; directive, and ignore the null dot operator. Do you think that's a reasonable way to approach the null dot operator?

Also, I see that you've added an #ecs; command. Would it be worthwhile to provide a compiler flag that inserts an #ecs; node automatically? If so, should it be on by default?

Anyway, those just some of my thoughts. I'm in the middle of my examination period, so I probably won't be implementing these features right away.

I am (slowly) working toward a heap-to-stack optimization pass for Flame, though. The general idea is to perform escape analysis, and then re-write some reference type values as references to value type instances, which would in turn make them susceptible to the scalar replacement of aggregates optimization. ecsc could benefit from this optimization, as well.

[qwertie]

Currently the ?. operator macro is incomplete, since it was written before it was possible to use variable declarations in expressions, so things like Foo?.Bar currently become Foo != null ? Foo.Bar : null and Foo is evaluated twice. This is not hard to fix, but it's also not known to a macro whether Foo is a property or a local variable or field (the first can only be evaluated once, the second could be evaluated twice safely in all cases - well maybe not if volatile). In the macro I would use the "lowercase rule" that I've used earlier (because creating temporary variables unnecessarily produces ugly output) but the proper EC# compiler could make a better decision than LeMP.

My thought is, though, that I could improve the ?. macro right now and then later when ecsc starts to mature, its code could be moved into ecsc - or alternately, moved into a second macros-pass of ecsc that runs after symbol tables are built, like in Nemerle.

There is a second operator we need to worry about: ?[...]. I've been wondering whether to change the precedence of ?. to match .; right now A.B?.C.D parses as (A.B)?.(C.D), because we need to prevent D from being evaluated in case A.B is null, and giving ?. a lower precedence made this relatively easy to do. However I don't see an equivalent strategy to handle A?[B].C.D - we can't parse it like A?[B].(C.D), treating ?[] as a ternary operator, because the . before C is not part of the ?[] operator (e.g. A?[B][C.D] or A?[B](C.D) or A?[B]?.C.D also exist)... so I am puzzled about how to handle the ?[] operator, and whatever strategy is used for ?[] can probably be used for ?. too.

I thought briefly about whether LeMP should wrap the whole input file in a node with a predictable name like #compilationUnit so that a macro could be written to process the whole file implicitly, without being requested, but that leaves the question of how to enable or prevent such a macro from running.

But it makes sense for ecsc to always add #ecs to the beginning of the file, because it is not relevant to the user whether the EC# features implemented by ecsc are implemented by a macro or by the "real" compiler.

Scalar replacement of aggregates? To be much use, that would have to occur after function inlining, wouldn't it? Usually one calls methods on one's aggregates. Good luck on your exams!

[jonathanvdc]

Maybe the ?. could be implemented by introducing yet another macro that may or may not introduce a new variable, and is implemented separately by ecsc and LeMP. Suppose it were called #evalOnce. Then we could write the following:

{
    #evalOnce(A.B, tmp, { print(tmp); print(tmp); });
    #evalOnce(A.b, tmp, { print(tmp); print(tmp); });
}

LeMP would then translate that as:

{
    { var tmp = A.B: print(tmp); print(tmp); }
    { print(A.b); print(A.b); }
}

And ecsc could instead just create a new temporary variable every time. Unless A.B were a type, of course, in which case tmp would become an alias for A.B.

Having a macro that can abstract over LeMP's casing voodoo and ecsc's symbol tables might also be useful for other macros, I suppose.

I'm not sure how to handle ?[] operator's precedence, either. Perhaps you could try giving ?. and ?[] the same precedence as ., and then inspect chains of member access/call/indexing operators, looking for null conditional operators.

Yes, exactly. Inlining is performed just before scalar replacement of aggregates at the moment. Together, those two passes can deliver fairly large performance improvements, and I'd like to make some class instances zero-cost as well.

Aggressive optimizations - which Roslyn lacks completely - could also be an extra selling point for using ecsc and, by extension, EC#.

Good luck on your exams!

Thanks. :)

[qwertie]

Looks like this issue should have been closed years ago.