Set up a public CI for the project

[dungpa]

(I know this is being considered. Just want to make it explicit for discussion.)

Setting up a public Continuous Integration would benefit both maintainers and contributors. It reduces workloads for maintainers in the long term. Once the solution works on CI, it's likely to work on arbitrary dev machines as well.

Since we have VS integration part, I suggest to use AppVeyor CI. It has been used on multiple VS extensions e.g. VsVim, Visual F# Power Tools with success.

I see a few immediate problems at the moment.

• We require quite a lot of prerequisite software https://github.com/Microsoft/visualfsharp/wiki/Required-Software.
• There isn't a single-click build and test process. It should be addressed by WIP: Create a FAKE Build #15.
• VS integration build and testing requires extra work. But since VS unit tests use mocks; we don't need real VS instances in testing. We can make this work on CI by supplying correct dependencies.

Any ideas/thoughts?

[forki]

The first thing we need to do is activate appveyor for visualfsharp project. We need someone with enough appveyor account and enough github rights for this.
I can then try to activate the build on #15 and see where it fails.
We probably also need some help from a appveyor person. Do you know a person?

[sergey-tihon]

I know that VS 2015 CTP 5 + SDK image is coming to AppVeyor CI.
I believe that we can do CI for F# 4.0 option and install all other prerequisites from Chocolatey using FAKE or discuss our needs directly with AppVeyor Team

@forki I think that you can activate AppVeyor for you fork of visualfsharp first

[dungpa]

If the decision is to go for AppVeyor, we probably need a paid account. The free account has time limit which might not be enough for building and testing the compiler.

I think VS 2015 CTP 5 + SDK would be good enough for now. We can always ask @FeodorFitsner (He has been very responsive and helpful on my AppVeyor-related questions).

[forki]

I think we should start with the basic unittests and qa tests. If have a CI build which does this it would already be a big win. We could then discuss how to extend it to the VS integration tests and how to get the money for the paid account. I think that this could be something for the FSSF (at least if Microsoft isn't paying).
/cc @ReedCopsey

[tpetricek]

This certainly sounds like something that would be perfectly aligned with the core mission of the F# Foundation and I'm pretty sure we would be able to find money for that. But I agree with @forki that we should start with a simpler thing - and then see how to best scale it to the full thing.

[tpetricek]

(That said, there might be some formal trickery as this repo is owned by Microsoft and not by the F# Foundation... but I'm not sure to what extent that matters.)

[forki]

Does appveyor allow us to run as admin? We need to ngen the proto-compiler.

[dungpa]

Yes, it does e.g. http://help.appveyor.com/discussions/questions/394-installing-a-database-from-mdf

[forki]

As a start I activated AppVeyor for my own fork and #15: https://ci.appveyor.com/project/SteffenForkmann/visualfsharp

[ReedCopsey]

I agree this would be a great thing to get setup - and the FSSF could potentially fund it if required.

Would it make more sense to do this on the open edition first? It probably should be on both repositories anyways... That might give us a cleaner way to get this setup, and also to show that it's not disruptive when we approach getting this added to the MS repository.

I can start the conversation and do some research into what would be required on the FSSF/Microsoft side of this.

@dungpa From what I can tell, the 40 minute build limit applies to all versions of AppVeyor - even the paid editions. The main difference there would potentially be better specs on the underlying system, so it might run a bit more quickly, but we're going to have to be careful about build times with any plan...

[forki]

First successful build: https://ci.appveyor.com/project/SteffenForkmann/visualfsharp/build/0.0.1.1

[dungpa]

@forki Great.

I wonder what F# team think about the whole thing.

[KevinRansom]

Steffen,

can you let us know what was needed to enable this for now. In the longer term we were thinking of Travis, because of it's unix support. But I think this is a great first step.

[forki]

There is a "getting started" at http://www.appveyor.com/docs

Basically it's just logging in and activating the project. But someone with Ms github account needs to do it.

And then you need to merge #15

[forki]

Regarding Travis: I think we need both. Will try to add minimal Travis build next week

[latkin]

We are certainly in favor of having a CI set up. .NET on *nix is a real thing that we will need to support sometime in the future, so our CI solution should not be Windows-only.

Last I heard from @jaredpar is that Roslyn is investigating Jenkins, as it supports both Windows and Linux. Appveyor is Windows-only, Travis is Linux/Mac-only.

[KevinRansom]

Yes, Lincoln is correct, Jenkins is what Jared is investigating.

[forki]

As I said: we need both. Projectscaffold-like projects already made very
good success with the combination of appveyor and Travis. It's much better
working than all the other combinations we tried before. And we tried a lot
(incl teamcity, Jenkins, cc.net)
On Jan 18, 2015 9:45 PM, "Lincoln Atkinson" notifications@github.com
wrote:

We are certainly in favor of having a CI set up. .NET on *nix is a real
thing that we will need to support sometime in the future, so our CI
solution should not be Windows-only.

Last I heard from @jaredpar https://github.com/jaredpar is that Roslyn
is investigating Jenkins, as it supports both Windows and Linux. Appveyor
is Windows-only, Travis is Linux/Mac-only.

—
Reply to this email directly or view it on GitHub
#53 (comment)
.

[KevinRansom]

Thank you for taking the time to investigate this Steffen, it is very important and necessary work, we are very grateful for your expertise and effort. As you can tell we are still learning what is important and how to prioritize it. We know we want to support platforms other than windows, together we will figure out which solutions make sense for this our project, these investigations of yours are a vital part of that effort.

Once again thanks

Kevin

[forki]

Just to clarify: the other CI servers also work very well. The thing that made appveyor and Travis make work very well for us is the configuration via yaml files. This allows us as a community to collaboratively maintain the CI process. People can send pull requests to fix CI issues. That's super important and I think other CI servers are also trying to support this configuration style.

[dungpa]

I think an important thing is to get the build infrastructure in place. We need a transparent and objective way to build and test the compiler. I would suggest to start with free versions of AppVeyor/Travis first. They're simple to configure and use. Moving to Jenkins (if desired) should be no brainer when build infrastructure is ready.

[jaredpar]

Correct, we are investigating Jenkins at the moment. It should in fact be working now but it appears there was a server hiccup over the weekend. Once it gets fixed on Monday though Roslyn will be on Jenkins.

[forki]

Is that a public Jenkins instance or are you hosting this? How is the config/maintenance handled? One issue for us on self-hosted CI servers was the time-zone difference between europe and the US.

[jaredpar]

@forki I'm unsure of the details of the current hosting. Meeting with the owners today to better understand their setup and how we can integrate into it.

Why would time zone be an issue?

[forki]

Thanks for keeping us updated. I'd love to hear what Roslyn is planning.

Regarding time-zone-issue: we used the public Teamcity service on codebetter.com for a very long time (IIRC since the beginning of FAKE (over 5 years ago)). The people over there were very nice and helpful, but from time to time you need an admin since the server went down or project settings needed adjustments. These points were very painful since there was no way for us to move forward. We needed to wait (at least 8h) until someone on the other side of the ocean rebootet the service and things like that.

Since we are using travis and AppVeyor these issue are completely gone:

1. These CI servers are maintained around the clock and are working in some cloud service which is basically always available. If the service goes down, then someone will immediatly work on it.
2. The configuration is done via yml files. So everybody can try different configs in their own fork. We don't need project admins to press buttons on a UI which normal contributors don't see. From time to time we still need a server admin to install VS SDK or things like that. But that case is very rare.

[theoy]

+1 to @jaredpar 's comment about sharing with Roslyn and CoreFx. Microsoft has been quite supportive of the efforts as well, and if we combine efforts it improves the likelihood that we can get buyin for premium services if we pick a solution that has that business model. Super glad to see this happening.

[dungpa]

@theoy Thanks for your comment. All things considered, sharing Jenkins infrastructure with Roslyn and CoreFx seems like a good way to move forward.

[Andrea]

I ll bring that up to the FSSF board :D
I also think there are a bunch of CI providers that wouldn't mind
supporting this project, again this is something the board can probably
chase up

Andrea
www.roundcrisis.com

On 30 January 2015 at 18:08, Steffen Forkmann notifications@github.com
wrote:

We now have appveyor builds which is awesome. But builds are waiting for
hours to start. I wonder if the FSSF can sponsor a pro account.
http://www.appveyor.com/pricing
I know it's not decided if appveyor is the way to go, but for now it's the
best thing we have.

/cc @ReedCopsey https://github.com/ReedCopsey @tpetricek
https://github.com/tpetricek @ryanriley https://github.com/ryanriley

—
Reply to this email directly or view it on GitHub
#53 (comment)
.

[ctaggart]

I'd like to see AppVeyor used for this Microsoft/visualfsharp project. The great thing about AppVeyor is that all of us who fork and contribute to this project can have our own builds setup in seconds. Microsoft is already an AppVeyor customer listed at www.appveyor.com. Since Microsoft owns this repository, it would be best if they setup the appropriate account.

However, I understand that could take a while. It would be great if FSSF could sponsor something in the meanwhile. FSSF currently uses the fsgit AppVeyor account to build the fsharp repo.
https://ci.appveyor.com/project/fsgit/fsharp

I'm not sure if it is possible, but it would be great if the FSSF could add the webhook for this repository so that builds would be at:
https://ci.appveyor.com/project/fsgit/visualfsharp

Even better would be if the FSSF's account was fsharp instead of fsgit. I'll contact AppVeyor and see what the options are for FSSF.

/cc @FeodorFitsner

http://help.appveyor.com/discussions/questions/927-builds-for-f-software-foundation-fssf

[Andrea]

Hi there

Just an update, I put this up to the board and it looks like this is going ahead :D. We are checking what plan we are getting, but it's looking very good.
Of course, there will be better worded information soon, but I thought it would be good to update.

Great news on Monday FTW :D

[ctaggart]

That is great! AppVeyor confirmed that we could setup an fsharp account and get 50% off.

To get faster builds you'd need "Pro" plan which is $29.50/month (50% off) for OSS projects. It includes 1 concurrent job and each additional concurrent job for OSS is $20/month.

Sure, you can have additional fsharp account for that with Pro plan assigned.

If FSSF sponsors an AppVeyor Pro account for CI builds, it is simple to setup and requires minimal coordination.

1. FSSF would upgrade an existing account or create a new one.
2. FSSF would add the visualfsharp project in AppVeyor.
   
3. FSSF would give Microsoft (@KevinRansom) the project webhook.
   
4. Microsoft would update the GitHub project settings with the updated webhook.

CI builds would then be faster and hopefully concurrent at:
https://ci.appveyor.com/project/fsharp/visualfsharp/
instead of slow at:
https://ci.appveyor.com/project/KevinRansom/visualfsharp-radou/

Happy to help out in setup.

[FeodorFitsner]

Thanks, Cameron! One note here - you should add "GitHub" project, not Git. If there are no sufficient permissions to add webhook that that step will be skipped (so you can add it later).

"Git" is for "generic" self-hosted Git repos and it requires a different webhook format.

Let me know if you have any questions.

[FeodorFitsner]

Also, to order "Pro FOSS" plan the following link should be used: https://ci.appveyor.com/plan/order/oss_pro_monthly

[forki]

❤️

[ctaggart]

@FeodorFitsner I need some clarification. How do you add a GitHub project that you haven't forked? If forking is required, do you recommend creating a GitHub account specifically for this setup?

[FeodorFitsner]

Right, I see. OK, fork it, add it and then edit "Repository name" on General tab of project settings in AV to point parent repo. Works perfect for public projects.

[ReedCopsey]

I believe this is now setup - I will email @KevinRansom with details for the webhook.

[ctaggart]

@ReedCopsey Where can we chat briefly?

[ctaggart]

I was wrong in the steps outlined above. Here are updated steps:

1. Either upgrade the fsgit AV account or create a new fsharp AV account. Which did you do? Upgrading fsgit may be easier.
2. Connect the AV account to a GH account. fsgit is already connected to fsgit.
3. Fork/clone the visualfsharp project in GH.
4. In AV project settings, simply change the GitHub repository to be Microsoft/visualfsharp.

[ReedCopsey]

@ctaggart Sent you contact info.

[ReedCopsey]

Followup: Microsoft upgraded their account, so the current build is now using a paid account. Build times are dramatically improved.

[forki]

[dungpa]

@FeodorFitsner Would you mind to install ActiveState Perl on VS 2015 CTP5 images? We need it for running tests (see #169 (comment)). Thanks.

[Andrea]

lol :D ... the memories

Andrea
www.roundcrisis.com

On 2 February 2015 at 21:25, Steffen Forkmann notifications@github.com
wrote:

[image: tumblr_inline_n4eqevcgie1raprkq]
https://cloud.githubusercontent.com/assets/57396/6009479/61ae1dee-ab2a-11e4-92a8-12b8c65fceea.gif

—
Reply to this email directly or view it on GitHub
#53 (comment)
.

[FeodorFitsner]

Sure, will do that.

[FeodorFitsner]

The latest Active Perl has been installed to VS 2015 CTP image.

[latkin]

Thanks @FeodorFitsner! How do I use it? I've tried kicking off a new CI build at #216. This ultimately invokes perl from the PATH. It still seems to be msys perl, though, see the end of the log here.

Am I using the wrong image?

[FeodorFitsner]

You requested Perl on Visual Studio 2015 CTP image and we deployed it there only :)
Looking at https://github.com/Microsoft/visualfsharp/blob/fsharp4/appveyor.yml default image is used. You can try installing Perl with Chocolatey: https://chocolatey.org/packages/ActivePerl

[dungpa]

@FeodorFitsner Thanks for doing that. I was trying to use VS 2015 CTP image today, it is significantly slower than the default image (see https://ci.appveyor.com/project/KevinRansom/visualfsharp-radou/build/0.0.1.134 and #217 (comment)). Do you know why?

I tried to install ActivePerl via Chocolatey. Unfortunately, the package is broken and unmaintained (see comments in your link). Could you please install ActivePerl on the default image? It seems we have to stick with that image for a while.

[FeodorFitsner]

It's because once you select a custom image it goes to Azure, because currently Pro environment supports one image only. On Azure it takes time to provision a worker VM and it times slower.

[FeodorFitsner]

Will deploy Perl to Pro environment.

[dungpa]

@FeodorFitsner I see. Thanks a lot for the kind help.

[FeodorFitsner]

Active Perl 5.20.1 has been deployed to Pro environment. Should be in PATH by default.

[dungpa]

Wonderful. I've got a few green builds using Perl at #217. And they're fast 👍 .