MonoGame.Portable returns - now with Piranah's

[SimonDarksideJ]

Right, so I've been working with @dellis1972 on the PCL generator project and it looks like it is close to being done.
It now uses a PCL generator solution based on Mono.Cecil called Piranah (which Dean has also updated to run on a Mac)
Generator branch for MG can be found here: https://github.com/DDReaper/MonoGame/tree/PiranahPCLGenerator

The generator only uses the WindowsGL build as a template, which seems sufficient for the PCL to be used in a "Bait and Switch" style PCL as a core platform. Platform specific code should always remain in a Platform project (like iOS), where as all common "core" code should be used in the PCL.

Question is now how to provide this framework back in to the MonoGame project so it can be run in an automated fashion. In the branch above (using Protobuild as a template) I've created a new ThirdParty folder for Piranah and put the runtime exe and Dll's in there.
There is a commandline file to generate the project manually for test at the moment (need both the WindowsGL project built and an output folder in MonoGame.Framework/bin/PCL created first)

I've also started on a new MonoGame.Portable NuGet package, which is still a work in progress. In testing it needs more platform targets to work effectively.

So next steps?

[Nezz]

What's the point of having a PCL version of MonoGame? Is it a stripped-down version that allows you to turn a subproject into a PCL even if it uses something like Vector2?

[KonajuGames]

Used for the bait-and-switch technique where you use NuGet to add the PCL
version to your project and NuGet will substitute the platform-specific
version at build time.​ Useful for cross-platform solutions.

[tomspilman]

What's the point of having a PCL version of MonoGame?

I still feel like this personally. Even bait-and-switch requires the same or more csprojs to make it work.

But some people still like it which is why we are going to support it.

[SimonDarksideJ]

@Nezz PCL's solve two distinct problems with multi-platform dev.

1: No file linking / copying required, it's a single project. While both file linking / copying work fine, they do require maintenance to add new files or remove files from all projects over time.

2: Core compatibility. Any code that is in the PCL is guaranteed to be compatible with all platforms, no need for #IF's or other code to manage all the different platforms within the PCL. It will also restrict you from using code that will break some platforms, plus it also abstracts some basic features to ensure code is compiled specifically for a certain platform.

Further details here: http://darkgenesis.zenithmoon.com/monogame-building-portable-solutions/

But as Tom said, it's a choice. Some devs use it a lot, other veer away from it and prefer other methods of maintaining a cross-platform solution.

If you are only targeting one platform, then you never need use or worry about it.

[SimonDarksideJ]

Thanks for the support @KonajuGames & @tomspilman

So
1: what are your thoughts about how to incorporate this in to the build process.
2: Is using an exe (like protobuild) the correct way to go, or should the source project be used? (currently it is using my own fork which is a superset of the original piranha project, dean's code and mine.

Thoughts?

[tomspilman]

So next steps?

First lets wait to get 3.3 out before we go further with PCL. I plan for this to happen soon... in the next week.

After that we need to add the Piranah.exe to a folder in the dependencies repo....

https://github.com/Mono-Game/MonoGame.Dependencies

... then you can update the default.build script to trigger running Piranah.exe on the WindowsGL assemblies in question.

Finally we need to update the installer to install the PCL assembly and probably need to add a PCL template of some sort.

[dellis1972]

@tomspilman with regard to 3.3 , the VS templates will need updating for
Unified iOS. the Project Type Guids will need to change as well as the
assembly reference ('MonoTouch' -> 'Xamarn.iOS'). I can do a PR for that
this weekend if you like.

On 16 January 2015 at 08:11, Tom Spilman notifications@github.com wrote:

So next steps?

First lets wait to get 3.3 out before we go further with PCL. I plan for
this to happen soon... in the next week.

After that we need to add the Piranah.exe to a folder in the dependencies
repo....

https://github.com/Mono-Game/MonoGame.Dependencies

... then you can update the default.build script to trigger running
Piranah.exe on the WindowsGL assemblies in question.

Finally we need to update the installer to install the PCL assembly and
probably need to add a PCL template of some sort.

—
Reply to this email directly or view it on GitHub
#3405 (comment).

[tomspilman]

I can do a PR for that this weekend if you like.

Sounds good to me.

[Helikin]

We have also been working on a MonoGame PCL version. I would like to share my experience to show how useful a PCL version of MonoGame can be.

How it was created:
I created a new MonoGame.Framework.Portable project. I added all MonoGame C# files except some internal platform-specific files, like files from the platform-specific folders (e.g. "Android"). It also references the *.Null.cs classes instead of platform-specific classes (e.g. Texture2D.Null.cs instead of Texture2D.DirectX.cs).
Then I added #if PORTABLE statements until everything compiled. On some code paths I added NotImplementedExceptions. I think the total number of changed lines is around 100 and the process was relatively straightforward.
The result is a MonoGame.Framework.dll which contains the whole MonoGame API. PCL compatible stuff can be used, like Vector3. Platform-specific stuff can be called but will not work and throw an exception.
I want to clean up the project but didn't have the time to revisit this topic yet.

How it is used:
Game engine DLLs reference the PCL version of MonoGame. The executable game projects reference a platform-specific version of MonoGame -- standard bait-and-switch PCL.
That means, if you write a graphics engine using MonoGame, you only need one MyGraphicsEngine.csproj which produces one MyGraphicsEngine.dll which runs on the different platforms.
In our engine we have replaced 51 platform-specific C# projects by 9 PCL project.
We have tested it on Windows, Windows Store, Windows Phone, Linux, MacOS, Android and iOS.

Some more notes:
If I understand correctly, you are trying to separate a PCL core API of MonoGame from the platform-specific API? I do not see how this would work with bait-and-switch. For us it is essential that the MonoGame PCL version contains the whole common public API -- including platform-specific APIs, like GraphicsDevice or Texture2D.

I soon need to revisit our MonoGame version for tests on WP 8.1 and iOS 64-bit. Then I could also clean up our PCL version and prepare a pull request -- to let you compare our PCL approach with your PCL approach.

[SimonDarksideJ]

Interesting @Helikin , that is the same pattern as I used for the original PCL version of MG.
That is also made up of all the separate platforms and includes all the platform variant functions as such, also driven by hand.

This move is to create a generated "core" solution without the platform variances for now as in testing across several solutions, it made more sense to only use those variants on the platform rather than in the PCL (it just made it over complicated) .
As it's a working generator now (there were several attempts and different frameworks used) and we have the source to work from, if the trend moves back to a holistic PCL rather than a core one, it should be easy to update and change.

[CartBlanche]

Nice!

[Helikin]

@DDReaper What is the difference between the core PCL and the holistic/full PCL? Does the core PCL contain any stubs for platform-specific classes, like GraphicsDevice, GamePad, OcclusionQuery?

[SimonDarksideJ]

Right, probably my bad terminology there.
I only refer to it as Core because this PCL is generated from the WindowsGL project. In there the additional folder / classes for the other platforms aren't used (not even the Windows one's, this didn't used to be the case but times have moved on)
So WindowsDX, Windows 8 and Windows phone all use the "Window" folder in the MonoGame.Framework\Windows folder. This contains such things as WinForms classes, some Gamerservices and some extension methods for the keyboard. So these would not be in the base PCL lib.
Android and iOS have similar platform based classes for the AndroidGameWindow, iOSGameView, etc.

The only concern or gap I potentially see is that the Compass and Accelerometer classes are still separate. This would need addressing in the future. SO at present you would need to abstract this in your Platform project and "send" it to the PCL code. (also allows for special exceptions per platform)

You can test this generation of the package here - https://www.dropbox.com/s/797nxyy5vwsotlh/MonoGame.Portable.3.2.3-alpha.nupkg?dl=0

[Helikin]

Ah, I understand. I think we are talking about the same API set and the Core API is all we really need.

I dropped the PCL DLL from the package into our projects and it seems to work fine. Great job!

[Eversor]

This Bait and Switch approach sounds really good for complex multi-csproj games.
The biggest problem i see is that the main idea is to share csproj between platforms, but the only thing really shared between them is the cs files. References and configurations are likely to be different on production games.

I'll definitely give it a go on my next game, and see what happens.

[SimonDarksideJ]

With bait and switch I've had between 90% 100% shared code between multiple platforms.
Only things that are truly specific for the platform (such as Cortana integration on WP, google play on Android, etc)
But this will also enable more "plug-in" type frameworks for MonoGame as well.

Plan on doing more education material in this area once it's up and running again.

P.S. the existing MonoGame-portable package has been used quite extensively already, this is just the final move to bring it under the MG banner fully.

[realsaraf]

Thanks @DDReaper , I have used PCL from your shared nupkg. It works perfectly. Since my game is quite and does not used XNA extensively current state does the job for me and i might as well go to production with this unless i face any issues.

[dellis1972]

Can I request a PCL & Nuget for the MonoGame.Framework.Content.Pipeline assembly. I just finished off a XS template for a Pipeline extension project, a nuget would fix the references :)

[SimonDarksideJ]

Will look to creating that @dellis1972

[craftworkgames]

I just wanted to pipe in here and say that this is the point of using PCL's:

How it is used:
Game engine DLLs reference the PCL version of MonoGame. The executable game projects reference a platform-specific version of MonoGame -- standard bait-and-switch PCL.
That means, if you write a graphics engine using MonoGame, you only need one MyGraphicsEngine.csproj which produces one MyGraphicsEngine.dll which runs on the different platforms.
In our engine we have replaced 51 platform-specific C# projects by 9 PCL project.
We have tested it on Windows, Windows Store, Windows Phone, Linux, MacOS, Android and iOS.

I don't think the value of this should be under-estimated. This issue bothered me so much that I stopped using MonoGame completely and created my own game framework from scratch. I'm not suggesting people use my framework over MonoGame. I'm first to admit that my framework is not even close to being as complete as MonoGame.

However, I just wanted to point out that there is a better way. It is 100% possible to create a game framework that uses PCL's at the core and makes multi-platform development an absolute pleasure. I'm not even going to take credit for this, LibGDX has been doing it this way for years with great success.

Even though I've got my own game framework separated from MonoGame I've still very much love to see MonoGame go this way. It would make it possible to write various extensions to MonoGame much more easily and I could take the work I've done in my own framework and combine it with MonoGame to give everyone the best of both worlds.

As it stands, writing any kind of MonoGame extension requires that the extension is recompiled for every supported platform. This really sucks, and stands in the way of some really great things happening in the MonoGame community (like physics, GUI libraries, particle systems, etc).

[nkast]

I reference MonoGame.Framework.Portable (3.2.99.1-beta) in a Windows 10 Class library
and I got an error that InstancePlayLimitException was not found.
Is it possible that Piranah strip it out of the assembly?

[dsaf]

@DDReaper, @tomspilman, @KonajuGames Would it be possible to merge this and release the installer and packages?

[SimonDarksideJ]

Once it's ready for release. a little ways to go yet 💃

[tomspilman]

@DDReaper @dellis1972

So what remains to get this ready to include?

[dsaf]

@DDReaper Cool! Are you working on this at the moment? Cannot wait! 😌

[dsaf]

Interesting:

http://blogs.msdn.com/b/mvpawardprogram/archive/2015/06/18/demystifying-pcls-net-core-dnx-and-uwp-redux.aspx

One of the most recent changes to NuGet, and the .NET ecosystem by extension, is support for the dotnet TFM. The meaning of dotnet wasn’t clear at first and as reflected in my earlier blog post, it seemed like it was the new target for the “new” portable .NET packages being published to NuGet and consumed by DNX and UWP. The reality isn’t quite like that but is far more interesting. Rather than dotnet representing a particular target like netcore45, dnxcore5 or net46, it really means “I’m compatible with any targets that my dependencies are, check those.” It gets NuGet out of the platform guessing game and instead walks the dependency graph.

[craftworkgames]

I haven't read the full post yet but this is seems like excellent news for PCL's. Awesome.

[dsaf]

@tomspilman @DDReaper @KonajuGames

Can't it be something as simple as taking all common types and putting them into a separate library? It's literally a handful of types that give the most re-usability benefits. Take a look at these quality packages:

Only uses Vector3: https://github.com/Robmaister/SharpNav

Only uses BoundingBox and Vector2: https://github.com/netonjm/ChipmunkSharp

It's a good design decision anyway. Look at the Entity Framework 7 that Microsoft guys are building from scratch: https://github.com/aspnet/EntityFramework/tree/dev/src

Core is the core: https://github.com/aspnet/EntityFramework/blob/dev/src/EntityFramework.Core/EntityFramework.Core.csproj

Specific platforms are separate:
https://github.com/aspnet/EntityFramework/blob/dev/src/EntityFramework.SqlServer/EntityFramework.SqlServer.csproj
https://github.com/aspnet/EntityFramework/blob/dev/src/EntityFramework.Relational/EntityFramework.Relational.csproj

And few things that need to be reused without creating dependencies are linked:
https://github.com/aspnet/EntityFramework/tree/dev/src/Shared

[tomspilman]

@hach-que - It is possibly stuff we've defined for Win10 in the custom xslt files. We need to make sure we get these changes back into Protobuild.

[SimonDarksideJ]

@hach-que the protobuild generation works fine. But when you try to then open any solution, you get the error in VS.

[hach-que]

Ah yeah, it's because upstream I changed {ADDITIONAL_TRANSFORMS} to be <!-- {ADDITIONAL_TRANSFORMS} --> so that the XML file was still valid before Protobuild pre-processed it.

[hach-que]

I've sent through a PR which updates Protobuild to the latest version so this PR can be updated.

[dsaf]

Is it likely to be a part of the recently announced next release?

https://twitter.com/MonoGameTeam/status/647384584741646336

[SimonDarksideJ]

OK, so moved on with the .Core project after the ProtoBuild update, however no further along it seems.
I can generate the MonoGame.Core PCL along side the other projects, everything is fine, however, the library projects won't reference the PCL.
Changes are here: https://github.com/DDReaper/MonoGame/tree/develop.core
Here's the log with shows the project building and even referenced from the libs, it just doesn't appear in the project. (https://www.dropbox.com/s/rhx96tk76eriwmo/MGBuildLog.txt?dl=0)

On evaluation with the test project (which works), it seems when you have a platform name (which is needed for MG) it ignores the project?

Any ideas @hach-que ?

[hach-que]

You need to change the platforms tag to include all platforms. The ForcePCL attribute will force it to be a PCL project even while under other platforms.

[hach-que]

Also you almost certainly do not need to include GUIDs in this new project, so you can (and should) omit that entire section.

[SimonDarksideJ]

Already have the ForcePCL property defined in the Build file for the library.

<?xml version="1.0" encoding="utf-8"?>
<Project Name="MonoGame.Framework.Core" Path="MonoGame.Framework" Type="Library" Platforms="PCL">
  <ProjectGuids>
    <Platform Name="PCL">44FDC7D0-CE78-4826-8F14-81C8A2455A00</Platform>
  </ProjectGuids>

  <!-- Common assembly references -->
  <References/>

  <Properties>

    <NoWarn>1591,0436</NoWarn>
    <MonoDevelopPoliciesFile>Build/MonoDevelopPolicies.xml</MonoDevelopPoliciesFile>

    <RootNamespace>Microsoft.Xna.Framework</RootNamespace>
    <RemoveXnaAssembliesOnWP8>false</RemoveXnaAssembliesOnWP8>
    <PlatformSpecificOutputFolder>True</PlatformSpecificOutputFolder>

    <CustomDefinitions>
      <Platform Name="PCL">PORTABLE</Platform>
    </CustomDefinitions>
    <Property Name="ForcePCL" Value="True" />

  </Properties>

[SimonDarksideJ]

Also note @hach-que that I took your ForcePCL test project, added the Platforms tag definitions to the <Project definitions, this also prevented your sample from working

As if the Platforms tag interferes with it's ability to reference the PCL.

If I remove the Platforms tag from the PCL lib here, it causes ProtoBuild to generate a project PER platform, which is obviously not desired. There should only be one PCL csproj generated.

[hach-que]

No there will always be a project per platform, because there's always a single platform for the entire project hierarchy that's generated. The ForcePCL option just forces it to be generated as a PCL project instead of the normal project for that platform type.

The resulting assembly is the same across all platforms, it's just the project file name that's different.

[hach-que]

When you set Platforms="PCL", this means that Protobuild will not generate or reference the project if the platform is "Windows". So you want to remove that Platforms tag entirely (it will still be a PCL project though).

[SimonDarksideJ]

I agree with your thinking @hach-que however the output doesn't meet expectations.
A SINGLE core PCL project that can be shared across all the platforms is what is required, however by removing the Platforms property results in separate PCL projects being generated for each and every platform:

In further testing, it's also NOT generating the projects as PCL's. I've even altered the .build file to match the ForcePCL example completely and it's no longer generating a PCL?
Any ideas
Here's the updated .build definition

<?xml version="1.0" encoding="utf-8"?>
<Project Name="MonoGame.Framework.Core" Path="MonoGame.Framework" Type="Library">

  <Properties>
    <Property Name="ForcePCL" Value="True" />
  </Properties>
  <References>
    <Reference Include="System" />
  </References>

  <Files>

If you have a minute, feel free to download my branch fork and run protobuild yourself to see the result.
https://github.com/DDReaper/MonoGame/tree/develop.core

Could it be that when a project that HAS a platform assigned, is overriding the ForcePCL flag? That's the only difference I can see between MonoGame's implementation and your test sample.

[SimonDarksideJ]

As a side note, as MonoGame shared it's assemblyinfo across all projects, the following Custom definition is needed on the PCL project

    <CustomDefinitions>
      <Platform>PORTABLE</Platform>
    </CustomDefinitions>

Would this still be available or would it need it's own?

[hach-que]

If MonoGame isn't using the latest XSLT files, then ForcePCL may not be doing anything.

All this flag does is force the project to be generated as a PCL project; it doesn't change the platform name. This is a limitation of the way Protobuild works; there is one platform active during generation, and this platform name is constant between all project generation. It's not possible for one project to have a different platform because it introduces significant complexities in determining the path for other project references (especially when considering external projects as well).

However with the ForcePCL attribute, all of those MonoGame.Framework.Core.*.csproj projects will produce exactly the same output. Because C# projects are considered temporary under Protobuild, their naming or existence is not an issue; when the solution is opened and built, you will get the correct result.

When distributing the binary, you can pick any of the assemblies generated from those MonoGame.Framework.Core projects for distribution; they are all identical.

[hach-que]

I just checked the XSLT, and you'll need to do:

<CustomDefinitions>
    <Platform Name="Android">PORTABLE</Platform>
    <Platform Name="iOS">PORTABLE</Platform>
    <Platform Name="Linux">PORTABLE</Platform>
    <Platform Name="MacOS">PORTABLE</Platform>
    <Platform Name="Ouya">PORTABLE</Platform>
    <Platform Name="PCL">PORTABLE</Platform>
    <Platform Name="PSMobile">PORTABLE</Platform>
    <Platform Name="Windows">PORTABLE</Platform>
    <Platform Name="Windows8">PORTABLE</Platform>
    <Platform Name="WindowsGL">PORTABLE</Platform>
    <Platform Name="WindowsPhone">PORTABLE</Platform>
    <Platform Name="WindowsPhone81">PORTABLE</Platform>
    <Platform Name="Web">PORTABLE</Platform>
</CustomDefinitions>

[tomspilman]

@hach-que - Actually it would be good for you to pull the changes we've made in our XSLTs into Protobuild. These are mainly just changes to support Windows 10...

https://github.com/mono/MonoGame/commits/develop/Build/GenerateProject.CSharp.xslt
https://github.com/mono/MonoGame/commits/develop/Build/GenerateSolution.xslt

... you probably want some or all of these changes in Protobuild. I just haven't had time to submit a PR to Protobuild for this myself.

Anyway... if Protobuild has Windows 10 support we could probably drop the XSLTs we have.

[SimonDarksideJ]

I had a look to see if I could recreate the changes you've made @tomspilman and create a PR for protobuild. But unfortunately there seems to be too much variation between the current xslt's used in MonoGame to the masters currently in Protobuild.

Any change @hach-que you can cherry pick the Win 10 fixes, then I can simply take the new masters as part of this change?

Granted, I know you are both up to your eyeballs with stuff at the moment.

[SimonDarksideJ]

*edit
Ok, after taking a chill pill and working through the XSLT definitions, I managed to locate all the UAP updates added by @tomspilman and submitted a PR to protobuild @hach-que
https://github.com/hach-que/Protobuild/pull/130

I've replaced the MG versions with the new updates and tested against the develop branch and noted no issues in the Windows 10 build.

My only reservation (which I've noted in the Protobuild commit) is that I'm not sure about naming Windows 10 as UAP, since the proper Windows 10 definition is UWP. Should UAP still be used or should it update to avoid confusion?

[hach-que]

Also this is tangentially related to Protobuild in MonoGame, but you might find it useful - I added post-build hooks recently which allow you to perform steps after building. Potentially MonoGame could have a project that executes a Pipeline build for all content project files in a project after the assembly is built, so that content is automatically built for anyone using MonoGame with Protobuild. All this requires is added PostBuildHook="True" to a project definition (for a console app) and it will run as a post-build hook for other projects in the solution. This is an example of a post-build hook project: https://github.com/hach-que/Protoinject/blob/master/Build/Projects/Protoinject.FactoryGenerator.definition (scroll across to see the PostBuildHook part).

[SimonDarksideJ]

Looks like the Core split isn't going to make the 3.5 milestone as the Protobuild update has been removed from the 3.5 milestone.

[SimonDarksideJ]

Right #4282 has been merged now, so the .Core project should now work.
Then I can get the PCL package done with the platform only bits. Tick tock.

[SimonDarksideJ]

now that the Protobuild updates are merged, reverified new .Core project PCL configuration, separating non-platform specific code in to a separate PCL lib (except Linux).
PR #4339 submitted.
Next step once this is merged is to have the platform specific bait and switch pcl to meet demand

[tomspilman]

Replacing this issue with one that is more easy to understand #4689. Let me know if I missed something.