Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

upstream main merge 2021 11 29 #6

Closed
wants to merge 324 commits into from

Conversation

joshpeterson
Copy link

am11 and others added 30 commits November 10, 2021 18:22
* Delete unused mono_unbox_int

* Fix gcc step display name

* Match command invocation order
Create a new `runtime-coreclr superpmi-asmdiffs` pipeline that runs SuperPMI asmdiffs for every change in the JIT directory.

The diffs are run on two platforms: Windows x64 and Windows x86. Linux, and Arm64 and Arm32, asm diffs are done using cross-compilers, as follows:

| Platform | Asm diffs |
| -- | -- |
| Windows x64 | win-x64, win-arm64, linux-x64, linux-arm64 |
| Windows x86 | win-x86, linux-arm |

The resulting summary .md files are uploaded into the pipeline artifacts, one .md file per platform (so, one for the Windows x64 runs and one for the Windows x86 runs). The results are also displayed in "Extensions" page of the AzDO pipeline.

The runs take about 50 minutes to complete (assuming not much waiting for machines).

The asm diffs pipeline is similar to the "superpmi-replay" pipeline, except:
1. It determines what an appropriate baseline JIT would be based on the PR commit and how it merges with the `main` branch. Given this, it downloads the matching baseline JITs from the JIT rolling build artifacts in Azure Storage.
2. It clones the `jitutils` repo and builds the `jit-analyze` tool, needed to generate the summary .md file.
3. It downloads and adds to the Helix machine payload a "portable" `git` installation, as `git diff` is used by `jit-analyze` for analyzing the generated .dasm files of the diff.
4. It collects all the various summary.md files into one per platform on which the runs are done, and publishes that to the artifacts and the `Extensions` page.
5. It only does one replay (asmdiffs) run, not one for each of a set of multiple stress modes.

As part of this implementation,
a. The `azdo_pipelines_util.py` was renamed to `jitutil.py`, and a lot of utility functions from superpmi.py were moved over to it. This was mostly to share the code for downloading and uncompressing .zip files. (There is a slight change to the output from the `superpmi.py download` command as a result.) However, I also moved a bunch of simple, more general helpers, for possible future sharing.
b. `jitrollingbuild.py download` can now take no arguments and download a baseline JIT (from the JIT rolling build Azure Storage location), for the current enlistment, to the default location. Previously, it required a specific git_hash and target directory. There is similar logic in superpmi.py, but not quite the same.
c. The `superpmi.py --no_progress` option was made global, and applied in a few more places. This was necessary because `superpmi.py asmdiffs` will download a coredistools binary from the JIT Azure Storage if one isn't found in the Core_Root folder.

Fixes dotnet#59445
… dotnet/hotreload-utils dotnet/llvm-project (dotnet#61365)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Ankit Jain <radical@gmail.com>
Previously the Directory.Build.targets script used to automatically
infer that OutputType=Library without a CLRTestKind implies
SharedLibrary. This is however hard to consolidate with the planned
test merging - as the SDK script set OutputType=Library by default,
we need the combination Library+(implicit)BuildAndRun to indicate
the "new-style" [Fact]-based tests. For this reason I propose to
remove this automatic inference and manually fix the handful of tests
that are missing an explicit CLRTestKind=SharedLibrary property.
In light of this description we can theoretically remove the
OutputType=Library specification from all test projects but even if
we decide to do that, I believe it will be easier to do that as a
separate mechanical change, not as part of this relatively small
change that has a different purpose. Additionally in the one case
of the GitHub_22583 regression test, I removed the explicit setting
of GenerateRunScript=false because that's the default.

Fix OutputType=exe in OpenDelegate.csproj

I believe this was a pre-existing bug - previously, with the
special clause regarding SharedLibrary, the test just got silently
skipped because it was considered to be a shared library.

Thanks

Tomas
* coreclr/

Make sure integral types respect sign extension during widen operation for Invoke Array ctor.
Remove always false IsStructRequiringStackAllocRetBuf().

* mono/

Create macro define for SPAN_T.

* libraries/

Add tests for Reflection Binder type conversion support during Invoke.
Catch patterns like the one in dotnet#37904 where a trinary compare feeds a
binary compare.
* We were not recording precise info in inlinees except for at IL offset
  0 because most of the logic that handles determining when to attach
  debug info did not run for inlinees. There are no changes in what the
  EE sees since we were normalizing debug info back to the root anyway.

* Propagate debug info even further than just until rationalization, to
  make it simpler to dump the precise debug info. This means we create
  some more GT_IL_OFFSET nodes, in particular when the inlinee debug
  info is valid but the root info is invalid. This is currently
  happening for newobj IL instructions when the constructor is inlined.
  We generate two statements:
  GT_ASG(GT_LCL_VAR(X), ALLOCOBJ(CLS));
  GT_CALL(CTOR, GT_LCL_VAR(X))
  and the first statement ends up "consuming" the debug info, meaning we
  end up with no debug info for the GT_CALL, which eventually propagates
  into the inline tree. I have held off on fixing this for now since it
  causes debug info diffs in the data reported back to the EE.

  The additional nodes in LIR result in 0.15% more memory use and 0.015%
  more instructions retired for SPMI over libraries.

There is also a small fix in gtlist.h for GT_BFIZ when
MEASURE_NODE_SIZES is defined.

No SPMI diffs.
There are connectivity issues on some physical Android devices. We should disable the affected tests until the issue isn't resolved.

Ref dotnet#61343
…net#60966)

We reference entry #0 in the pinned plug queue even if there are no pinned plugs at all and thus the pinned plug queue contains left-over data from the mark phase.

The fix is to initialize saved_pinned_plug_index to a value that is invalid as a pinned plug queue index, and only use saved_pinned_plug_index as an index if  is valid.
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Don't run if the JIT-EE GUID has changed, since there won't be any collections
to download, so the downloads will fail.

Also update the superpmi-asmdiffs trigger comment.
… annotate individual methods as requiring AVX (dotnet#61259)

* Updating src/tests/Interop/PInvoke/Generics/GenericsNative.Vector* to annotate individual methods as requiring AVX

* Always use __m256i on XARCH
* XARCH: Remove redudant tests for GT_LT/GT_GE relops.

We can now optimize cases such as `(x + y < 0)` or `for (int x = v; x >= 0; x--)`
using the flag tracking logic during the emit stage. Notably, cases that
would generate...

```
add     reg0, reg1
test    reg0, reg0
jge     LABEL
```

now transform to

```
add     reg0, reg1
jns     LABEL
```

This transform is valid for signed GE and signed LT only.

* Add a few asserts related to flag reuse optimizations.
* Moving Narrow to implemented using SIMDAsHWIntrinsic

* Moving Widen to implemented using SIMDAsHWIntrinsic

* Fix some handling of Narrow/Widen hwintrinsics

* Ensure that Vector.Widen is still treated as an intrinsic

* Fixing NI_VectorT128_WidenUpper on ARM64 to actually call gtNewSimdWidenUpper
Measure browser app start times, 2 measurements implemented.

First to measure till the JS window.pageshow event, second to measure
time when we reach managed C# code.

Example ouput:

    | measurement | time |
    |-:|-:|
    |                    AppStart, Page show |   108.1400ms |
    |                AppStart, Reach managed |   240.2174ms |
* Environment.SystemPageSize returns cached value

* we are no longer shipping MS.IO.Redist, so we can use Array.MaxLength directly

* we are no longer shipping MS.IO.Redist, there is no need for File to be partial

* we are no longer shipping MS.IO.Redist, there is no need for FileInfo to be partial

* there is no need for .Win32.cs and .Windows.cs file anymore
elinor-fung and others added 26 commits November 24, 2021 14:19
* Reorganize libs.native

* Fix darwin build

* Fix wasm-win32 build

* Fix pipelines path evaluation
* New globalThis.getDotnetRuntime method, which takes runtimeId, essentially index of the runtime on the page. It returns the DotNetPublicAPI object of the instance {MONO, BINDING, Module, RuntimeId, RuntimeBuildInfo }.
* Change the debugger to use the getDotnetRuntime() function when talking to the runtime on the page via CDP.
* Use getDotnetRuntime() in unit tests, with runtimeId is zero as there is only one runtime in tests.
* We add optional &runtimeId=0 to the initial URL which opens DebuggerProxy, so that MonoProxy could be created for specific runtime on the page.
* Moved mono_wasm_add_dbg_command_received, mono_wasm_debugger_log and mono_wasm_trace_logger out of C macro into typescript
* Introduced RuntimeBuildInfo: { ProductVersion, Configuration } into DotNetPublicAPI
* [interp] Improve logging on mobile devices

Use a single g_print in bulk for every instruction. Otherwise, the instruction ends up being displayed on multiple lines.

* [interp] Remove hack for nint/nfloat

These structures are valuetypes, but their mint_type is a primitive. This means that a LDFLD applied on the type would have attempted to do a pointer dereference, because it saw that the current top of stack is not STACK_TYPE_VT. This was fixed in the past by passing the managed pointer to the valuetype rather than the valuetype itself, against the normal call convention, which lead to inconsistencies in the code.

This commit removes that hack and fixes the problem by ignoring LDFLD applied to nint/nfloat valuetypes in the first place.
…ls on SUBST'd drive (dotnet#59850)

* Re-enable test that verifies DriveInfo.VolumeLabel setter fails on SUBST'd drive
Co-authored-by: Tratcher <Tratcher@users.noreply.github.com>
Co-authored-by: James Newton-King <james@newtonking.com>
* Add test for ignoring stepping into hidden function.

* Corrected test to match Console App behaviour.

* Corrected test logic.

* Applied Thays's fix so that step into would work as step over in a hidden method.

* Revert debugging changes.

* Test for Debugger.Break().

* Col fix + checing the location after command execution.

* Correct value of VisibleMethod entry line.

* Move the breakpoint to the line 846.

* Create tests that match behaviour of Console App.

* Draft of Debugger.Break behaviour correction.

* Draft: 2 pauses on hidden method with Debugger.Break inside.

* Changed required behaviour to 2 pauses on the hidden method call.
…net#62023)

This fixes a new warning generated by trimming some apps which was introduced in dotnet#54056.
The `ComVisibleAttribute` in this case is referenced, but if it's removed it doesn't change functionality in any way.
* Avoid switching GC mode in COM's HasValidTarget

Co-authored-by: Jan Kotas <jkotas@microsoft.com>
Co-authored-by: JamesNK <JamesNK@users.noreply.github.com>
in high memory load situations, when we had to wait for a BGC to finish, we switched to preemptive mode which means GCs could have occurred and replenished the budget. but when we come back from the wait, we use the previous decision on whether the budget was exceeded or not. this causes us to trigger GC incorrect and you would see GCs triggered when the allocated bytes are tiny.
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Premek Vysoky <premek.vysoky@microsoft.com>
Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com>
* Fix miscellaneous typos

* Cleanup trailing whitespaces
```sh
# git remote add dotnet https://github.com/dotnet/runtime && git pull --rebase dotnet main

if uname 2>/devnull | grep -q Darwin; then
    space=" "
fi

git show --name-only --pretty="" HEAD...dotnet/main |\
    xargs -I{} sh -c "test -f {} && sed -i$space'' 's/[[:space:]]*$//' {}"
```
…62003)

* mono_runtime_delegate_invoke wrapper with MONO_ENTER_GC_UNSAFE/MONO_EXIT_GC_UNSAFE;

mono_runtime_delegate_invoke wrapper with MONO_ENTER_GC_UNSAFE/MONO_EXIT_GC_UNSAFE;

* Update object.c
The CSE logic maintains a custom hashtable implementation.

It triggers a resize at the same time as adding a new CSE,
but forgets the hash of the new CSE needs to be updated from
its pre-resize value. Failing to do so can lead to losing some
CSEs, though not in a correctness-impacting way.
Otherwise we can end up not seeing the loop has memory havoc.

Also added an assert that will prevent this issue from arising in the future.
…et#60054)

* CI matrix change: add Windows.Server.Core.20H2 (dotnet#59572)

* Add Windows.Server.Core.20H2 to CI matrix

* add windows server 20H2

* Remove 20H2 from PR

* Remove 20H2 from PR run

* change helix queue for Windows.Server.Core.20H2
* Remove active issue.
* Use --web-server-use-cop.
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
@joshpeterson joshpeterson deleted the upstream-main-merge-2021-11-29 branch November 29, 2021 20:42
yuc434 pushed a commit that referenced this pull request Aug 8, 2022
* Initial implementation for contract customization

fix build errors

Move converter rooting to DefaultJsonTypeInfoResolver so that it can be used standalone

Fix ConfigurationList.IsReadOnly

Minor refactorings (#1)

* Makes the following changes:

* Move singleton initialization for DefaultTypeInfoResolver behind a static property.
* Consolidate JsonSerializerContext & IJsonTypeInfoResolver values to a single field.
* Move reflection fallback logic away from JsonSerializerContext and into JsonSerializerOptions

* Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializerOptions.cs

* remove testing of removed field

Simplify the JsonTypeInfo.CreateObject implemenetation (#2)

* Simplify the JsonTypeInfo.CreateObject implemenetation

* Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Metadata/JsonTypeInfoOfT.cs

* Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Metadata/JsonTypeInfoOfT.cs

Co-authored-by: Krzysztof Wicher <mordotymoja@gmail.com>

Co-authored-by: Krzysztof Wicher <mordotymoja@gmail.com>

Tests and fixes for JsonTypeInfoKind.None

TypeInfo type mismatch tests

Allow setting NumberHandling on JsonTypeInfoKind.None

test resolver returning wrong type of options

JsonTypeInfo/JsonPropertyInfo mutability tests

rename test file

Move default converter rooting responsibility behind DefaultJsonTypeInfoResolver (#3)

* Move default converter rooting responsibility behind DefaultJsonTypeInfoResolver

* address feedback

Add simple test for using JsonTypeInfo<T> with APIs directly taking it

fix and tests for untyped/typed CreateObject

uncomment test cases, remove todo

More tests and tiny fixes

Add a JsonTypeInfoResolver.Combine test for JsonSerializerContext (#4)

* Fix JsonTypeInfoResolver.Combine for JsonSerializerContext

* Break up failing test

Fix simple scenarios for combining contexts (#6)

* Fix simple scenarios for combining contexts

* feedback

JsonSerializerContext combine test with different camel casing

Remove unneeded virtual calls & branching when accessing Get & Set delegates (#7)

JsonPropertyInfo tests everything minus ShouldSerialize & NumberHandling

Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs

Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs

throw InvalidOperationException rather than ArgumentNullException for source gen when PropertyInfo.Name is assigned through JsonPropertyInfoValues

tests for duplicated property names and JsonPropertyInfo.NumberHandling

Add tests for NumberHandling and failing tests for ShouldSerialize

disable the failing test and add extra checks

disable remainder of the failing ShouldSerialize tests, fix working one

Fix ShouldSerialize and IgnoreCondition interop

Add failing tests for CreateObject + parametrized constructors

Fix CreateObject support for JsonConstructor types (#10)

* Fix CreateObject support for JsonConstructor types

* address feedback

Make contexts more combinator friendly (#9)

* Make contexts more combinator friendly

* remove converter cache

* redesign test to account for JsonConstructorAttribute

* Combine unit tests

* address feedback

* Add acceptance tests for DataContract attributes & Specified pattern (#11)

* Add private field serialization acceptance test (#13)

* tests, PR feedback (#14)

* PR feedback and extra tests

* Shorten class name, remove incorrect check (not true for polimorphic cases)

* Make parameter matching for custom properties map property Name with parameter (#16)

* Test static initialization with JsonTypeInfo (#17)

* Fix test failures and proper fix this time (#18)

* Fix test failures and proper fix this time

* reinstate ActiveIssueAttribute

* PR feedback and adjust couple of tests which don't set TypeInfoResolver

* fix IAsyncEnumerable tests

* Lock JsonSerializerOptions in JsonTypeInfo.EnsureConfigured()

Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com>

Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com>
mrvoorhe pushed a commit that referenced this pull request Jul 11, 2023
…tnet#87189)

This fixes a startup crash on Big Sur:

> error: * Assertion at /Users/runner/work/1/s/src/mono/mono/utils/mono-hwcap-arm64.c:35, condition `res == 0' not met

Because sysctl can't find some of these options:

    $ sysctl hw.optional.armv8_crc32
    hw.optional.armv8_crc32: 1
    $ sysctl hw.optional.arm.FEAT_RDM
    sysctl: unknown oid 'hw.optional.arm.FEAT_RDM'
    $ sysctl hw.optional.arm.FEAT_DotProd
    sysctl: unknown oid 'hw.optional.arm.FEAT_DotProd'
    $ sysctl hw.optional.arm.FEAT_SHA1
    sysctl: unknown oid 'hw.optional.arm.FEAT_SHA1'
    $ sysctl hw.optional.arm.FEAT_SHA256
    sysctl: unknown oid 'hw.optional.arm.FEAT_SHA256'
    $ sysctl hw.optional.arm.FEAT_AES
    sysctl: unknown oid 'hw.optional.arm.FEAT_AES'

Full stack trace:

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1
  * frame #0: 0x0000010ef37560 libmonosgen-2.0.dylib`monoeg_assertion_message
    frame #1: 0x0000010ef375cc libmonosgen-2.0.dylib`mono_assertion_message + 32
    frame #2: 0x0000010ef40d6c libmonosgen-2.0.dylib`mono_hwcap_arch_init + 544
    frame #3: 0x0000010ef54bd8 libmonosgen-2.0.dylib`mono_hwcap_init + 72
    frame #4: 0x0000010ee14dc0 libmonosgen-2.0.dylib`parse_optimizations + 52
    frame #5: 0x0000010edbed48 libmonosgen-2.0.dylib`mono_init
    frame #6: 0x0000010ee18968 libmonosgen-2.0.dylib`mono_jit_init_version
    frame #7: 0x0000010f48a300 libxamarin-dotnet-debug.dylib`xamarin_bridge_initialize + 216
    frame #8: 0x0000010f4900a4 libxamarin-dotnet-debug.dylib`xamarin_main + 376
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet