New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Umbrella] Compilers should be deterministic: same inputs generate same outputs #372

Open
gafter opened this Issue Feb 10, 2015 · 22 comments

Comments

@gafter
Member

gafter commented Feb 10, 2015

This is an umbrella issue for making the Roslyn compilers deterministic. See also Open Issues for Determinism.

There are a few issues:

  • #223 Anonymous types should be output in a deterministic order.
  • #360 The string heap should be output in a deterministic order.
  • #803 The order of symbols produced from the DataFlowAnalysis API should be deterministic.
  • #926 Produce PDBs with deterministic GUIDs and timestamps by default. This requires a CLR spec change; see dotnet/coreclr#1615. The CLR spec has been agreed internally and implemented in Roslyn, but the new spec has not been published and this is not enabled by default.
  • #949 Produce only relative paths in caller file name arguments and in the PDB
  • #1228 Netmodules should generate per-assembly unique names for PrivateImplementationDetails
  • #1319 VB Anonymous types should generate per-assembly unique names for netmodules
  • #1428 C# should mangle module name uniquely to produce anonymous types.
  • #1430 C# PrivateImplementationDetails should not have "." in its name
  • #1440 Ordering of synthesized delegates in metadata should be deterministic
  • #1506 Diagnostics should be returned in a deterministic order
  • #2184 Make the command-line compiler capable of generating a reference assembly
    • #17612 Refine what is in reference assemblies and what diagnostics prevent generating one
  • #2303 Add support for /features:deterministic for msbuild (fixes timestamp and MVID)
  • #4171 Attributes are sometimes emitted in nondeterministic order
  • #4172 VB static locals emitted to fields in nondeterministic order
  • #4221 Roslyn parses double values differently on x32 vs x64
  • #4578 Auto-generated assembly version is nondeterministic
  • #5070 Document what compiler inputs affect its deterministic output
  • #7111 VB interfaces not emitted it a deterministic order across partial types
  • #7112 VB shared field initializers should be emitted in source order across partial types
  • #7262 Constant folding uses wrong precision
  • #7595 Small method body cache nondeterministic (see also #17052)
  • #8703 Review usage of random guid in Emit/ErrorType.cs
  • #9813 Need a mechanism to cause file-relative PDB path to be included in the assembly.
  • #10321 Document compiler /determinism effect on generated assembly
  • #10858 Enumerate files deterministically when compiling *.cs or *.vb
  • #11015 Symbol.Locations has indeterministic ordering of locations between compilations with same inputs
  • #11990 The order in which type forwarders are emitted to ExportedType table is non-deterministic
  • #17121 Document compiler inputs for determinism (for cacheability)
  • #23020 Generation of the GetHashCode method of anonymous types is not deterministic
  • #30439 Produce the same NaN bits on different platforms

@gafter gafter self-assigned this Feb 10, 2015

@gafter gafter added this to the 1.1 milestone Feb 12, 2015

@gafter gafter modified the milestones: 1.0-rc2, 1.1 Feb 27, 2015

@gafter gafter added 3 - Working and removed 2 - Ready labels Feb 27, 2015

@gafter gafter changed the title from Investigate making module IDs deterministic to Compilers should be deterministic: same inputs generate same outputs Mar 5, 2015

@gafter gafter added 2 - Ready and removed 3 - Working labels Mar 6, 2015

@gafter gafter modified the milestones: 1.0 (stable), 1.0-rc2 Mar 9, 2015

@gafter gafter changed the title from Compilers should be deterministic: same inputs generate same outputs to [Umbrella] Compilers should be deterministic: same inputs generate same outputs Mar 19, 2015

@gafter gafter modified the milestones: 1.1, 1.0 (stable) Mar 31, 2015

@gafter gafter added 3 - Working and removed 2 - Ready labels Apr 13, 2015

@gafter gafter added 2 - Ready and removed 3 - Working labels May 5, 2015

@martinsuchan

This comment has been minimized.

martinsuchan commented Nov 29, 2017

Just my 2 cents - the msbuild flags /deterministic and /pathmap looks like already implemented based on the checklist, but they are not documented anywhere. Any chance to fix it?
MicrosoftDocs/visualstudio-docs#361

@jcouv

This comment has been minimized.

Member

jcouv commented Nov 29, 2017

Thanks @martinsuchan for pointing out those issues.

Documenting /pathmap is tracked by dotnet/docs#1800 (do chime there to voice your interest).

I filed another documentation issue just now for /deterministic at dotnet/docs#3828

@ghost

This comment has been minimized.

ghost commented Mar 31, 2018

Why is /deterministic flag an optional thing in many compilers; is there a penalty / cost attached can be avoided without csc /deterministic? Otherwise, if it is all about bringing goodness then does it has to be optional, hidden behind a flag?

@jcouv

This comment has been minimized.

Member

jcouv commented Mar 31, 2018

@kasper3 I think the only reason why determinism isn't the default is to avoid surprising customers, who have been using the compiler (without determinism) for a long time.
Deterministic assemblies have strange timestamps, which would be surprising to many if the default was changed. Also determinism prohibits the use of wildcard in assembly version.
That's why customers have to make an explicit choice to turn determinism on.

@svick

This comment has been minimized.

Contributor

svick commented Mar 31, 2018

@kasper3 @jcouv BTW, /deterministic is the default for .Net Core SDK projects, so over time, it should be the default for more and more new projects.

@jcouv

This comment has been minimized.

Member

jcouv commented Mar 31, 2018

@svick Good point.
Filed dotnet/project-system#3438 to update the desktop templates (so at least newly created projects can used determinism).

@ghost

This comment has been minimized.

ghost commented Mar 31, 2018

@jcouv, the still-unchecked items in the first post can be revisited. Some of them are closed issues (some are discussion-going-nowhere open issues).

@martinsuchan

This comment has been minimized.

martinsuchan commented Apr 3, 2018

Wondering, are there any plans to make UWP app builds deterministic by default? Timestamps and wildcards in assembly versions are not really important when publishing apps to Store.
Last time I was testing deterministic builds for UWP apps it was not possible to produce identical packages because WinMDExp and MakePri.exe tools from the Windows SDK build pipeline don't support producing deterministic outputs yet.
Also since Microsoft Store uses differential updates for downloading new packages, having deterministic builds by default could save tons of Store traffic basically for free.

@jcouv

This comment has been minimized.

Member

jcouv commented Apr 3, 2018

Tagging @MichalStrehovsky @sergiy-k for UWP question.

@MichalStrehovsky

This comment has been minimized.

Member

MichalStrehovsky commented Apr 4, 2018

This came up in the context of .NET Native in the past (#23456) but I couldn't find the owners of the tools in question. Maybe @tarekgh would know for MakePri?

@tarekgh

This comment has been minimized.

Member

tarekgh commented Apr 4, 2018

@axelandrejs can help answering MakePri question.

@axelandrejs

This comment has been minimized.

axelandrejs commented Apr 4, 2018

MakePri is expected to produce identical output for identical input. Now, it is possible that some of the inputs change unexpectedly, since MakePri parses the file system. If you have two files that were produced over the same set of inputs but are not identical, can you share the files?

Note that if you are implementing your own build pipeline, we now have a programmatic equivalent to MakePri. See https://msdn.microsoft.com/en-us/library/windows/desktop/mt845690(v=vs.85).aspx . It avoids dependencies on the file system and as such takes out some of the potential non-determinism. Also, it's faster. We are developing a tool that supports specifying all the data needed to build a UWP resources file via a relatively simple XML, in case that would help.

@paulsapps

This comment has been minimized.

paulsapps commented Apr 13, 2018

Does anyone know if resgen.exe is deterministic?

@martinsuchan

This comment has been minimized.

martinsuchan commented Jul 23, 2018

@axelandrejs @tarekgh Reviving the discussion, MakePri does not produce deterministic outputs, nor WinMDExp, just checked again in VS 15.7.4. There is a simple repro solution available in #23456. Just build the solution twice in the same clean folder and extract all appxupload/appx files in it.
The expected result is identical content of both folders, the reality is differences in resources.pri and Winrt.Component.winmd.
Any chance to find the product owners of those tools and make them deterministic by default?

Files with different checksum:

\AppxBlockMap.xml
\AppxSignature.p7x
\ReproTest_1.0.0.0_x86.appx
\AppxMetadata\AppxBundleManifest.xml
\ReproTest_1.0.0.0_x86\AppxBlockMap.xml
\ReproTest_1.0.0.0_x86\AppxSignature.p7x
\ReproTest_1.0.0.0_x86\resources.pri
\ReproTest_1.0.0.0_x86\Winrt.Component.winmd
\ReproTest_1.0.0.0_x86\AppxMetadata\CodeIntegrity.cat

@tarekgh

This comment has been minimized.

Member

tarekgh commented Jul 23, 2018

@axelandrejs may be able to help better here.

@axelandrejs

This comment has been minimized.

axelandrejs commented Jul 23, 2018

The problem is not actually makepri. The problem are the XBF files, specifically App.xbf and MainPage.xbf. They are different between runs.

By default XAML files are embedded as BLOBs into the PRI files to improve performance. This means that if the XBF files change, you will see the difference in the resources.pri file. You can see for yourself. You can dump the contents of the resoures.pri file via the following command.

makepri dump /dt detailed /if [the resoures.pri file] /of [some .xml file name]. I did this for the two files you had in ReproTest.build1 and ReproTest.build2. You can see the the App.xbf contents below.

@LarryOsterman can speak to the .winmd differences.
@jevansaks can speak to the .xbf differences.

WEJGAJYBAABaAAAAAgAAAAEAAAB4AAAAAAAAAHYBAAAAAAAAegEAAAAAAAB+AQAAAAAAAIIBAAAAAAAAhgEAAAAAAAAzNjZCNDlDODVFRjdEOTIyQkRERTQ4RTA2MzY3MUJCMQAICAAzAGEAEiodUrgWAIzg/iADAQAAAAAAAAAAAAAAAwAAADkAAABoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAC8AdwBpAG4AZgB4AC8AMgAwADAANgAvAHgAYQBtAGwALwBwAHIAZQBzAGUAbgB0AGEAdABpAG8AbgAAACwAAABoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAC8AdwBpAG4AZgB4AC8AMgAwADAANgAvAHgAYQBtAGwAAAAPAAAAdQBzAGkAbgBnADoAUgBlAHAAcgBvAFQAZQBzAHQAAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAABAAAAAgAAAAEAAAAAAAAATgAAABIAAAAAAAADAQABAAAAeAADAgAFAAAAbABvAGMAYQBsAAsNAAAAUgBlAHAAcgBvAFQAZQBzAHQALgBBAHAAcAAXH4AaSIALQgIAAAAAIQ==

VERSUS

WEJGAJYBAABaAAAAAgAAAAEAAAB4AAAAAAAAAHYBAAAAAAAAegEAAAAAAAB+AQAAAAAAAIIBAAAAAAAAhgEAAAAAAAAzNjZCNDlDODVFRjdEOTIyQkRERTQ4RTA2MzY3MUJCMQAICAAAAAAAHQT6JAASAIpTAHkAcwB0AGUAbQAuAE8AAwAAADkAAABoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAC8AdwBpAG4AZgB4AC8AMgAwADAANgAvAHgAYQBtAGwALwBwAHIAZQBzAGUAbgB0AGEAdABpAG8AbgAAACwAAABoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAC8AdwBpAG4AZgB4AC8AMgAwADAANgAvAHgAYQBtAGwAAAAPAAAAdQBzAGkAbgBnADoAUgBlAHAAcgBvAFQAZQBzAHQAAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAABAAAAAgAAAAEAAAAAAAAATgAAABIAAAAAAAADAQABAAAAeAADAgAFAAAAbABvAGMAYQBsAAsNAAAAUgBlAHAAcgBvAFQAZQBzAHQALgBBAHAAcAAXH4AaSIALQgIAAAAAIQ==

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment