Make F# resources work more like C# #3352

nosami · 2017-07-19T16:03:24Z

Given a resource in a folder named AFolder/AResourceFile.txt in a project with
a root namespace of fsharpresources, then the following command line arg
should be passed to the compiler

--resource:AFolder/AResourceFile.txt,fsharpresources.AFolder.AResourceFile.txt

whereas currently it just passes the name of the file. This also fixes passing
a LogicalName to the resource.

Fixes #1050 &&
#922

I looked at how Roslyn was doing this here

Given a resource in a folder named `AFolder/AResourceFile.txt` in a project with a root namespace of `fsharpresources`, then the following command line arg should be passed to the compiler ``` --resource:AFolder/AResourceFile.txt,fsharpresources.AFolder.AResourceFile.txt ``` whereas currently it just passes the name of the file. This also fixes passing a `LogicalName` to the resource. Fixes dotnet#1050 && dotnet#922 I looked at how Roslyn was doing this [here](https://github.com/dotnet/roslyn/blob/6847f1e5a909395aae9456e8f366cbf4deb86b69/src/Compilers/Core/MSBuildTask/ManagedCompiler.cs#L739) es. Lines starting

nosami · 2017-07-19T16:20:51Z

This also fixes https://bugzilla.xamarin.com/show_bug.cgi?id=57689 and https://bugzilla.xamarin.com/show_bug.cgi?id=37971

dsyme · 2017-07-19T16:32:34Z

@nosami This has been a huge source of pain historically - really glad you have sorted this out

Could you also submit to fsharp/fsharp and we'll check what the extra testing there says (there is testing enabled in that repo that is not enabled here)

nosami · 2017-07-19T16:34:11Z

It took me most of today to work that out :)

Will submit to fsharp/fsharp... I tested with that repo locally, then applied the PR here :)

KevinRansom · 2017-07-19T18:29:36Z

@dsyme what is the nature of the extra testing?

nosami · 2017-07-19T19:14:34Z

Fwiw, I ran the Xamarin tests that build each F# template in VS for Mac via msbuild with this change applied. All passed.

dsyme · 2017-07-19T20:33:07Z

@KevinRansom I think it is

the compilation of tests/projects against an installation of F# on Mono
a matrix of Mono installations (nightly, weekly, specific versions) automated by Travis https://github.com/fsharp/fsharp/blob/master/.travis.yml
OSX testing https://github.com/fsharp/fsharp/blob/master/.travis.yml

The problems with Linux/OSX testing in this repo are

All we do here is build. https://github.com/Microsoft/visualfsharp/blob/master/netci.groovy#L57.
There is duplication in build.sh/build.cmd in this repo, which is unsustainable.

For now I've personally run out of time w.r.t. on the repo unification work and have to focus on language design and research work.

dsyme · 2017-07-19T20:34:03Z

Fwiw, I ran the Xamarin tests that build each F# template in VS for Mac via msbuild with this change applied. All passed.

@nosami Great to know you have these tests. Resources are used nearly always in UI. Do the tests also run the resulting apps?

KevinRansom · 2017-07-19T20:35:10Z

@don Syme<mailto:Don.Syme@microsoft.com> You aint kidding about the Cambridge and QA test suites. But nothing gets achieved if nothing gets done on them.

nosami · 2017-07-19T20:35:35Z

@dsyme The tests just download nuget packages, check that msbuild works, and that there are no squiggles in the editor - https://github.com/mono/monodevelop/blob/master/main/external/fsharpbinding/MonoDevelop.FSharp.Tests/TemplateTests.fs

dsyme · 2017-07-19T20:57:53Z

@nosami I'm concerned that the generated names of resources may be wrong - this usually shows up as a runtime failure in some "GetResource" call

dsyme · 2017-07-19T20:58:40Z

@nosami You have one failure:

1) Failed : Tests.ProjectSystem.Miscellaneous.CreateFSharpManifestResourceName
[("Abc.resources", "Abc.resources"); ("Bar.de.resx", "Bar.de");
 ("Bar.resx", "Bar"); ("Xyz\Baz.ru.resx", "Xyz.Baz.ru")]<>[("Abc.resources", "RootNamespace.Abc.resources");
 ("Bar.de.resx", "RootNamespace.Bar.de"); ("Bar.resx", "RootNamespace.Bar");
 ("Xyz\Baz.ru.resx", "RootNamespace.Xyz.Baz.ru")]
at <StartupCode$VisualFSharp-Unittests>.$Tests.ProjectSystem.Miscellaneous.CreateFSharpManifestResourceName@444.Invoke(String file) in D:\j\workspace\release_ci_pa---3f142ccc\vsintegration\tests\unittests\Tests.ProjectSystem.Miscellaneous.fs:line 454
at UnitTests.TestLib.Utils.FilesystemHelpers.DoWithTempFile[a](String filename, FSharpFunc`2 f) in D:\j\workspace\release_ci_pa---3f142ccc\vsintegration\tests\unittests\TestLib.Utils.fs:line 73
at Tests.ProjectSystem.Miscellaneous.CreateFSharpManifestResourceName() in D:\j\workspace\release_ci_pa---3f142ccc\vsintegration\tests\unittests\Tests.ProjectSystem.Miscellaneous.fs:line 444

you might need to adjust the test, since it looks like

RootNamespace is now being taken into account
The name of the resource in a subdirectory has changed from Xyz\Baz.ru.resx to Xyz.Baz.ru (likewise Xyz\Baz.ru.resx to RootNamespace.Xyz.Baz.ru)

We will also need guidance about whether this is a breaking change (for Mono and/or Windows)

nosami · 2017-07-19T21:18:09Z

Well, the generated names may well be different now, this is definitely a breaking change. But, I think it's a good breaking change - it looks as if it's never worked correctly.

I think that the test is wrong and needs to be changed to match the code. The new names look correct to me.

nosami · 2017-07-19T21:25:56Z

As far as mono goes, this change is needed. Some things in Xamarin iOS just plain don't work https://bugzilla.xamarin.com/show_bug.cgi?id=57689 being a prime example. I would rather have that work and take the fallout from the change.

dsyme · 2017-07-19T21:29:55Z

As far as mono goes, this change is needed. Some things in Xamarin iOS just plain don't work https://bugzilla.xamarin.com/show_bug.cgi?id=57689 being a prime example. I would rather have that work and take the fallout from the change.

Yes, I'm inclined to believe that. I don't have access to that specific bug though

@cartermp and @KevinRansom will need to make the call about the breaking change. Are the scenarios simply

"I have resources in my project and they are in a sub-folder"?
"I have a RootNamespace element in my project file and use some resources"?

Is there anything we could do to moderate the breaking change? e.g. only activate it if some other setting is passed in, and set that in the Xamarin projects and all future templates?

nosami · 2017-07-19T21:31:38Z

The main issue is that you can't have the same filename in multiple folders. This happens frequently for Xamarin, e.g. the same icon but in different sizes - they get put in subfolders but the filename is the same.

It would be far too difficult to try and work against that with F# specific hacks.

dsyme · 2017-07-19T21:35:54Z

It would be far too difficult to try and work against that with F# specific hacks.

I totally agree, I want to see this fixed once-and-for-all.

But can we make this conditionally activated in some way by a <UseCorrectResourceNames>true</UseCorrectResourceNames> setting or something? Or even by using new-style project files? Or some Xamarin project file setting? We just need to make sure this is not breaking for existing projects.

matthid · 2017-07-19T21:52:27Z

is this actually breaking? Was using folders ever officially supported? It definitely never felt like it was.

dsyme · 2017-07-19T21:54:02Z

is this actually breaking? Was using folders ever officially supported? It definitely never felt like it was.

@matthid Yes, I believe you could do it and inspect the assembly for the resource names generated. Also this now respects <RootNamespace>Foo</RootNamespace> which was previously ignored

nosami · 2017-07-19T21:55:11Z

Yes, I'm inclined to believe that. I don't have access to that specific bug though

Do you have a bugzilla account? I thought this was a public bug

As this is quite risky (even though I feel it's right), I could apply this patch to mono only. That way only mono / xamarin users would be fixed... but then Windows users would still be affected

I feel like that the correct solution would be to have a switch that reverted this behaviour.

dsyme · 2017-07-19T21:59:37Z

As this is quite risky (even though I feel it's right), I could apply this patch to mono only. That way only mono / xamarin users would be affected... but then Windows users would still be affected

I think we want this fix to apply for Xamarin-on-Visual-Studio-for-Windows too. Inconsistency in Xamarin tooling between Windows and OSX and Linux would be bad

I feel like that the correct solution would be to have a switch that reverted this behaviour.

Are you sure we can't activate it explicitly? That would be the safest route. We don't want existing F# code breaking.

KevinRansom · 2017-07-19T22:00:32Z

@dsyme

It's interesting that this is not a compiler fix, but instead a fix in the build task modification just changing what's passed to the compiler.

It seems that we could and should enable this as the default approach. And add a buildtask option to use the old behavior to enable as a quick fix for anyone whose app breaks.

I don't think there is a lot of risk of making this change, if we have a simple fallback.

Kevin

nosami · 2017-07-19T22:01:48Z

Are you sure we can't activate it explicitly? That would be the safest route. We don't want existing F# code breaking.

For sure, we can do this. But it feels (to me) like adding a switch to make it behave like it should behave is wrong - there should be a switch to enable the opposite.

dsyme · 2017-07-19T22:10:34Z

@nosami Could you also review these notes please? This is where I documented all the variations in behaviour a while back. At the time I didn't know how to fix the problem at root cause. Also those notes don't cover RootNamespace. (The behaviour as documented there was when using XBuild on Linux - XBuild caused havoc, and we are at last ridding ourselves of that, which is I think what's allowing us to make progress on this at last. We should check again with MSBuild on Linux)

Perhaps write up a new version of all those scenarios too, and we need to check behaviour on both Windows and OSX.

I don't think there is a lot of risk of making this change, if we have a simple fallback.

This is not right. Do we really want upgrading to VS2017 15.4 or the latest Xamarin package to just give random "missing resource" failures on startup of applications?

Silently changing the names of generated resources has busted, among other things, the compiler packages and apps on multiple occasions in the past.

For the compiler package the resources suddenly had wrong embedded names.
For the Xamarin/MonoDevelop addin the names were inconsistent on Windows and Mono, so it depended where you compiled things.

In both cases the failures only manifest at runtime, and sometimes we detected the problem only once we upgraded to use the (broken) compiler package.

I absolutely want this problem to cease, but I'm 100% convinced that this has the potential to be a risky breaking change to compilations that use resources - including the two cases where I have used them - since most people end up putting resources in directories sooner or later - and most project files have RootNamespace set silently.

We would only consider this safe if we were 99.9% sure that no one was doing that - but we know that they are. We were nervous about fixing this blindly when we first came across the whacky behaviour (and unimplemented RootNamespace behaviour) in F# 3.0 timeframe. If we were nervous then, we should be even more careful now.

I really want this fixed, but we should

determine the exact scenarios where this changes existing behaviour and
if necessary, we have to make it opt-in and non-breaking for existing project files (it can be the default for new .NET SDK project files)

If we release this change then I'm fairly confident it will silently break multiple instances of existing enterprise code (resources tend to get used in enterprises more than data-scripting/ETL code). In that case we are really in the proverbial, as we will likely have to revert to old behaviour, and then we will have even more confusion about what the actual behaviour is. (I really don't want to be in a situation where @KevinRansom, @Pilchie, @cartermp, myself and others at Microsoft have to hot-patch the F# compiler when MrBigBank calls Microsoft and says that their apps are now crashing on startup now they've upgraded to latest VS2017...)

dsyme · 2017-07-19T22:29:24Z

@nosami Is the RootNamespace part of the fix necessary here? My understanding from your comment above is that it isn't?

BTW the steps in this bug indicate all that's required for a user to have done for her to be faced with a breaking change here:

Create a new F# Library project
Add a folder called "AFolder"
Add a text file to AFolder called "AResourceFile.txt"
Set the build action on that text file to embedded resource

That is a very common scenario. Many people will have embedded resources into their applications or libraries at some point. They will now get a silent "resource not found" or null pointer exception at runtime if they recompile. And VFPT has been allowing folder creation in Visual Studio since late 2014, and Xamarin/MonoDevelop have always allowed it.

I hate the fact that this was botched in F# 2.0 - and we have to dig ourselves out of the hole. But silently breaking existing code isn't the way out.

nosami · 2017-07-19T22:36:15Z

@dsyme: You know the risks far better than I do :) I know that this is super risky. Whatever you decide is fine with me :)

dsyme · 2017-07-19T22:43:22Z

@nosami The only thing I can think of is a new setting

<UseStandardResourceNames>true</UseStandardResourceNames>

(or some similar property) which we put in F# VS and Xamarin templates for old-style project files, and we always activate it for new-style project files. Any chance you could you draft that?

KevinRansom · 2017-07-19T23:01:51Z

@dsyme
We will make this the default for SDK projects. And for new projects created using VS.

dsyme · 2017-07-19T23:03:23Z

@dsyme We will make this the default for SDK projects. And for new projects created using VS.

Super.

7sharp9 · 2017-07-20T10:59:55Z

Being able to have multi dpi images in F# Android Applications is a must really, with resources being broken for so long its difficulty to recommend F# for android at all.

dsyme · 2017-07-20T11:07:00Z

Being able to have multi dpi images in F# Android Applications is a must really

Yes, it must be fixed

With ``` <UseStandardResourceNames>true</UseStandardResourceNames> ``` we get the following output ``` --resource:AFolder/AResourceFile.txt,fsharpresource.AFolder.AResourceFile.txt ``` With the value omitted or set to false, the behaviour is as it was previously. ``` --resource:AFolder/AResourceFile.txt ```

nosami · 2017-07-20T15:41:05Z

OK. I've added a UseStandardResources flag which is disabled by default.

With

    <UseStandardResourceNames>true</UseStandardResourceNames>

we get the following output

--resource:AFolder/AResourceFile.txt,fsharpresource.AFolder.AResourceFile.txt

With the value omitted or set to false, the behaviour is as it was previously.

--resource:AFolder/AResourceFile.txt

dsyme · 2017-07-20T16:03:05Z

src/fsharp/FSharp.Build/Microsoft.FSharp.Targets

@@ -89,24 +89,28 @@ this file.

        <CreateFSharpManifestResourceName
             Condition="'@(ResxWithNoCulture)' != '' AND '$(UsingXBuild)' == 'true'"
+             UseStandardResourceNames="$(UseStandardResourceNames)"


@KevinRansom Does anything need to be done for .NET SDK targets here?

@dsyme. I will update VS templates and the dotnet sdk.

@KevinRansom Is this fix in the latest .NET SDK or will it be in a future release? I cannot get it to work on my project, which looks like this:

<PropertyGroup> <OutputType>Exe</OutputType> <TargetFramework>netcoreapp2.0</TargetFramework> <RootNamespace>Generators</RootNamespace> <UseStandardResourceNames>true</UseStandardResourceNames> </PropertyGroup> <ItemGroup> <Compile Include="Program.fs" /> </ItemGroup> <ItemGroup> <EmbeddedResource Include="Templates\*.liquid" /> </ItemGroup>

If I use ildasm to look at the manifest, the resource doesn't have the prepended namespace:

.mresource public _TestClass.liquid

dsyme · 2017-07-20T16:04:05Z

@nosami Great. Could you also adjust that test that was failing to run in both modes and check each? And check that we have RootNamespace=null as a variation on the test too?

nosami · 2017-07-20T16:05:04Z

Sure - will do.

nosami · 2017-07-20T16:45:26Z

Writing the tests might not be so easy - I only have OSX on this laptop and VisualFSharp doesn't compile there. I can try and do it "blind". Or wait until next week when I am back on my main machine.

KevinRansom · 2017-07-21T06:00:43Z

src/fsharp/FSharp.Build/Microsoft.FSharp.Targets

@@ -89,24 +89,28 @@ this file.

        <CreateFSharpManifestResourceName
             Condition="'@(ResxWithNoCulture)' != '' AND '$(UsingXBuild)' == 'true'"
+             UseStandardResourceNames="$(UseStandardResourceNames)"


@dsyme. I will update VS templates and the dotnet sdk.

... to generate resource identifiers the same way that C# does. This allows having the same resource name in different folders. Follow on commit to dotnet/fsharp#3352 (comment)

msftclas added the cla-not-required label Jul 19, 2017

dsyme reviewed Jul 20, 2017

View reviewed changes

KevinRansom approved these changes Jul 21, 2017

View reviewed changes

KevinRansom merged commit f6049d5 into dotnet:master Jul 25, 2017

nosami deleted the fsharp-resource-fixes branch July 25, 2017 09:06

nosami mentioned this pull request Sep 4, 2017

Add UseStandardResourceNames msbuild setting... mono/monodevelop#2984

Merged

abelbraaksma mentioned this pull request Jun 29, 2018

Story on embedded resources seems to be rather outdated #5256

Closed

Make F# resources work more like C# #3352

Make F# resources work more like C# #3352

Conversation

nosami commented Jul 19, 2017 • edited

nosami commented Jul 19, 2017

dsyme commented Jul 19, 2017

nosami commented Jul 19, 2017

KevinRansom commented Jul 19, 2017

nosami commented Jul 19, 2017

dsyme commented Jul 19, 2017 • edited

dsyme commented Jul 19, 2017

KevinRansom commented Jul 19, 2017 via email • edited by dsyme

nosami commented Jul 19, 2017

dsyme commented Jul 19, 2017

dsyme commented Jul 19, 2017 • edited

nosami commented Jul 19, 2017

nosami commented Jul 19, 2017

dsyme commented Jul 19, 2017

nosami commented Jul 19, 2017

dsyme commented Jul 19, 2017

matthid commented Jul 19, 2017

dsyme commented Jul 19, 2017

nosami commented Jul 19, 2017 • edited

dsyme commented Jul 19, 2017

KevinRansom commented Jul 19, 2017

nosami commented Jul 19, 2017

dsyme commented Jul 19, 2017 • edited

dsyme commented Jul 19, 2017 • edited

nosami commented Jul 19, 2017

dsyme commented Jul 19, 2017

KevinRansom commented Jul 19, 2017

dsyme commented Jul 19, 2017

7sharp9 commented Jul 20, 2017

dsyme commented Jul 20, 2017

nosami commented Jul 20, 2017

dsyme Jul 20, 2017

Choose a reason for hiding this comment

KevinRansom Jul 21, 2017

Choose a reason for hiding this comment

ErikSchierboom Sep 11, 2017

Choose a reason for hiding this comment

dsyme commented Jul 20, 2017

nosami commented Jul 20, 2017

nosami commented Jul 20, 2017 • edited

KevinRansom Jul 21, 2017

Choose a reason for hiding this comment

nosami commented Jul 19, 2017 •

edited

dsyme commented Jul 19, 2017 •

edited

KevinRansom commented Jul 19, 2017 via email •

edited by dsyme

dsyme commented Jul 19, 2017 •

edited

nosami commented Jul 19, 2017 •

edited

dsyme commented Jul 19, 2017 •

edited

dsyme commented Jul 19, 2017 •

edited

nosami commented Jul 20, 2017 •

edited