Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LLVM AOT caches require SSE4.1-capable CPU #12182

Open
Nichtraucher opened this issue Dec 23, 2018 · 38 comments
Assignees

Comments

@Nichtraucher
Copy link

@Nichtraucher Nichtraucher commented Dec 23, 2018

Steps to Reproduce

  1. update to 5.18.0.225-0xamarin1+ubuntu1404b1

Current Behavior

update fails due to unresolved dependencies and other errors (see terminal output below)

Expected Behavior

errr...update goes through smoothly (as always)?

On which platforms did you notice this

Linux (Linux Mint 17.2 Rafaela (4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux). I'm using the repositories provided by mono.

Version Used:
Mono JIT compiler version 5.18.0.225
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
Interpreter: yes
LLVM: yes(600)
Suspend: preemptive
GC: sgen (concurrent by default)

Stacktrace

Terminal output here: https://pastebin.com/um5HptcP

I read in the chat that at least one other user seems to have the same/a similar issue, she posted her output here: https://0x0.st/sdBY.txt , hence my decision to post this report

any help is much appreciated!

@guydavis

This comment has been minimized.

Copy link

@guydavis guydavis commented Dec 23, 2018

Confirmed same problem with 5.18.0.225-0xamarin1+ubuntu1604b1

Workaround for Ubuntu 16.04 is to roll back to older 5.16 version which still works:

$ sudo apt remove mono* libmono*

Edit /etc/apt/sources.list.d/mono-official-stable.list to comment out stable and pin older version:
    #deb http://download.mono-project.com/repo/ubuntu stable-xenial main
    deb http://download.mono-project.com/repo/ubuntu xenial/snapshots/5.16.0.220 main

$ sudo apt update
$ sudo apt install mono-devel

Then was able to run Jackett, Sonarr, and Radarr again. I'll leave Mono pinned to 5.16 until this issue is resolved.

@simeon4110

This comment has been minimized.

Copy link

@simeon4110 simeon4110 commented Dec 23, 2018

I get stuck in a SIGKILL when attempting to remove the package:

=================================================================
Got a SIGILL while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Aborted (core dumped)
E: installing Assembly /usr/lib/cli/nunit.core.interfaces-2.6.3/nunit.core.interfaces.dll failed
E: Installation of libnunit-core-interfaces2.6.3-cil with /usr/share/cli-common/runtimes.d/mono failed
* Installing 1 assembly from libnunit-framework2.6.3-cil into Mono
Stacktrace:

/proc/self/maps:
40900000-40920000 rwxp 00000000 00:00 0
41322000-41332000 rwxp 00000000 00:00 0
55da82ca9000-55da830f2000 r-xp 00000000 08:22 13895047                   /usr/bin/mono-sgen
55da832f1000-55da832f8000 r--p 00448000 08:22 13895047                   /usr/bin/mono-sgen
55da832f8000-55da832fd000 rw-p 0044f000 08:22 13895047                   /usr/bin/mono-sgen
55da832fd000-55da8338e000 rw-p 00000000 00:00 0
55da84d2f000-55da84f9b000 rw-p 00000000 00:00 0                          [heap]
7f5ee0000000-7f5ee0021000 rw-p 00000000 00:00 0
7f5ee0021000-7f5ee4000000 ---p 00000000 00:00 0
7f5ee5c00000-7f5ee5d00000 rw-p 00000000 00:00 0
7f5ee5dc0000-7f5ee5dc1000 ---p 00000000 00:00 0
7f5ee5dc1000-7f5ee5dc2000 rw-p 00000000 00:00 0
7f5ee5dc2000-7f5ee5dca000 ---p 00000000 00:00 0
7f5ee5dca000-7f5ee5fc1000 rw-p 00000000 00:00 0
7f5ee5fc1000-7f5ee657d000 r-xp 00000000 08:22 14290069                   /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f5ee657d000-7f5ee677c000 ---p 005bc000 08:22 14290069                   /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f5ee677c000-7f5ee677d000 r--p 005bb000 08:22 14290069                   /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f5ee677d000-7f5ee677e000 rw-p 005bc000 08:22 14290069                   /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f5ee677e000-7f5ee67a8000 rw-p 00000000 00:00 0
7f5ee67a8000-7f5ee6bff000 r--p 00000000 08:22 14290071                   /usr/lib/mono/4.5/mscorlib.dll
7f5ee6bff000-7f5ee7bff000 rw-p 00000000 00:00 0
7f5ee7bff000-7f5ee7c00000 ---p 00000000 00:00 0
7f5ee7c00000-7f5ee8800000 rw-p 00000000 00:00 0
7f5ee885d000-7f5ee88aa000 r--p 00000000 08:22 14290003                   /usr/lib/mono/gac/Mono.Security/4.0.0.0__0738eb9f132ed756/Mono.Security.dll
7f5ee88aa000-7f5ee8b88000 r--p 00000000 08:22 13896927                   /usr/lib/locale/locale-archive
Memory around native instruction pointer (0x7f5ee60e48cf):
0x7f5ee60e48bf  20 48 09 d6 41 0f b6 53 05 48 c1 e2 28 48 09 f2   H..A..S.H..(H..
0x7f5ee60e48cf  66 41 0f 38 32 4b 06 66 0f 6f 05 72 75 49 00 66  fA.82K.f.o.ruI.f
0x7f5ee60e48df  0f 38 00 c8 66 48 0f 7e c8 66 48 0f 3a 16 ce 01  .8..fH.~.fH.:...
0x7f5ee60e48ef  48 09 c6 48 09 d6 41 0f b6 c2 41 0f b6 53 09 48  H..H..A...A..S.H

Native stacktrace:

        /usr/bin/mono(+0x129f8d) [0x55da82dd2f8d]
        /usr/bin/mono(+0x12a295) [0x55da82dd3295]
        /usr/bin/mono(+0xbfd2f) [0x55da82d68d2f]
        /usr/bin/mono(+0x414e8) [0x55da82cea4e8]
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f5ee91a3890]
        /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so(+0x1238cf) [0x7f5ee60e48cf]

Waiting for dumping threads to resume

Debug info from gdb:

[New LWP 17327]
[New LWP 17328]
[Thread debugging using libthread_db enabled]

This is a never ending loop.

@simeon4110

This comment has been minimized.

Copy link

@simeon4110 simeon4110 commented Dec 23, 2018

I managed to get it to purge by manually removing each package with dpkg.

@Thf772

This comment has been minimized.

Copy link

@Thf772 Thf772 commented Dec 23, 2018

Same problem on Ubuntu Bionic, a simple sudo apt-get upgrade is enough to reproduce the bug. The culprit is ca-certificates-mono=5.18.0.225-0xamarin1+ubuntu1804b1

Moreover, apt-get won't remove Mono automatically. It suggests apt --fix-broken install, which I tried and causes a near-endless loop of SIGILLs like @simeon4110 . Also, I can't install or upgrade any package at all, even if they are not related to Mono.

The following packages have unmet dependencies:
 ca-certificates-mono : PreDepends: mono-runtime-common but it is not going to be installed
                        Depends: mono-runtime (>= 3.0~) but it is not going to be installed
                        Depends: libmono-corlib4.5-cil (>= 4.0.0~alpha1) but it is not going to be installed
                        Depends: libmono-security4.0-cil (>= 3.0.6) but it is not going to be installed
 libglade2.0-cil : Depends: libmono-corlib4.5-cil (>= 4.0.0~alpha1) but it is not going to be installed
 libglib2.0-cil : Depends: libmono-corlib4.5-cil (>= 4.0.0~alpha1) but it is not going to be installed
                  Depends: libmono-system-core4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
                  Depends: libmono-system4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
 libgtk2.0-cil : Depends: libmono-cairo4.0-cil (>= 3.2.1) but it is not going to be installed
                 Depends: libmono-corlib4.5-cil (>= 4.0.0~alpha1) but it is not going to be installed
                 Depends: libmono-system-drawing4.0-cil (>= 3.0.6) but it is not going to be installed
 libnunit-console-runner2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
                                    Depends: libmono-system4.0-cil (>= 4.2.0) but it is not going to be installed
 libnunit-core-interfaces2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
                                     Depends: libmono-system4.0-cil (>= 4.2.0) but it is not going to be installed
 libnunit-core2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
                          Depends: libmono-system4.0-cil (>= 4.2.0) but it is not going to be installed
 libnunit-framework2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
                               Depends: libmono-system-xml4.0-cil (>= 4.2.0) but it is not going to be installed
                               Depends: libmono-system4.0-cil (>= 4.2.0) but it is not going to be installed
 libnunit-mocks2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
 libnunit-util2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
                          Depends: libmono-system-configuration4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
                          Depends: libmono-system-drawing4.0-cil (>= 3.0.6) but it is not going to be installed
                          Depends: libmono-system-runtime4.0-cil (>= 2.10.1) but it is not going to be installed
                          Depends: libmono-system-xml4.0-cil (>= 4.2.0) but it is not going to be installed
                          Depends: libmono-system4.0-cil (>= 4.2.0) but it is not going to be installed
 msbuild : Depends: mono-runtime (>= 3.0~) but it is not going to be installed
           Depends: libmono-corlib4.5-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-microsoft-build-framework4.0-cil (>= 3.6.0) but it is not going to be installed
           Depends: libmono-microsoft-csharp4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-componentmodel-composition4.0-cil (>= 3.0.6) but it is not going to be installed
           Depends: libmono-system-configuration4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-system-core4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-system-data4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-system-drawing4.0-cil (>= 3.0.6) but it is not going to be installed
           Depends: libmono-system-identitymodel4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-system-io-compression-filesystem4.0-cil (>= 3.2.1) but it is not going to be installed
           Depends: libmono-system-io-compression4.0-cil (>= 3.2.1) but it is not going to be installed
           Depends: libmono-system-net-http4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-numerics4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-reflection-context4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-runtime-serialization4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-system-security4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-servicemodel4.0a-cil (>= 3.2.3) but it is not going to be installed
           Depends: libmono-system-windows-forms4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-xaml4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-xml-linq4.0-cil (>= 3.0.6) but it is not going to be installed
           Depends: libmono-system-xml4.0-cil (>= 3.12.0) but it is not going to be installed
           Depends: libmono-system4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-windowsbase4.0-cil (>= 3.0.6) but it is not going to be installed
 referenceassemblies-pcl : Depends: mono-xbuild but it is not going to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
@simeon4110

This comment has been minimized.

Copy link

@simeon4110 simeon4110 commented Dec 23, 2018

If you purge manually it will work.

@lightway82

This comment has been minimized.

Copy link

@lightway82 lightway82 commented Dec 24, 2018

I can`t install and update other packages!

How I can solve this problem?

For any action this error occurs

mono-gac (5.18.0.225-0xamarin1+ubuntu1804b1) …

  • Installing 1 assembly from libglade2.0-cil into Mono
    Stacktrace:

/proc/self/maps:
408c0000-408e0000 rwxp 00000000 00:00 0
40d86000-40d96000 rwxp 00000000 00:00 0
5585b9373000-5585b97bc000 r-xp 00000000 08:06 529674 /usr/bin/mono-sgen
5585b99bb000-5585b99c2000 r--p 00448000 08:06 529674 /usr/bin/mono-sgen
5585b99c2000-5585b99c7000 rw-p 0044f000 08:06 529674 /usr/bin/mono-sgen
5585b99c7000-5585b9a58000 rw-p 00000000 00:00 0
5585bba2c000-5585bbc99000 rw-p 00000000 00:00 0 [heap]
7f92cc000000-7f92cc021000 rw-p 00000000 00:00 0
7f92cc021000-7f92d0000000 ---p 00000000 00:00 0
7f92d0600000-7f92d0700000 rw-p 00000000 00:00 0
7f92d07c1000-7f92d0d7d000 r-xp 00000000 08:06 787004 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f92d0d7d000-7f92d0f7c000 ---p 005bc000 08:06 787004 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f92d0f7c000-7f92d0f7d000 r--p 005bb000 08:06 787004 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f92d0f7d000-7f92d0f7e000 rw-p 005bc000 08:06 787004 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f92d0f7e000-7f92d0fa8000 rw-p 00000000 00:00 0
7f92d0fa8000-7f92d13ff000 r--p 00000000 08:06 529541 /usr/lib/mono/4.5/mscorlib.dll
7f92d13ff000-7f92d23ff000 rw-p 00000000 00:00 0
7f92d23ff000-7f92d2400000 ---p 00000000 00:00 0
7f92d2400000-7f92d3000000 rw-p 00000000 00:00 0
7f92d3091000-7f92d30de000 r--p 00000000 08:06 655588 /usr/lib/mono/gac/Mono.Security/4.0.0.0__0738eb9f132ed756/Mono.Security.dll
7f92d30de000-7f92d30df000 ---p 00000000 00:00 0
7f92d30df000-7f92d30e0000 rw-p 00000000 00:00 0
7f92d30e0000-7f92d30e8000 ---p 00000000 00:00 0
7f92d30e8000-7f92d3313000 rw-p 00000000 00:00 0
7f92d3313000-7f92d3723000 r--p 00000000 08:06 525494 /usr/lib/locale/locale-archive
Memory around native instruction pointer (0x7f92d08e48cf):
0x7f92d08e48bf 20 48 09 d6 41 0f b6 53 05 48 c1 e2 28 48 09 f2 H..A..S.H..(H..
0x7f92d08e48cf 66 41 0f 38 32 4b 06 66 0f 6f 05 72 75 49 00 66 fA.82K.f.o.ruI.f
0x7f92d08e48df 0f 38 00 c8 66 48 0f 7e c8 66 48 0f 3a 16 ce 01 .8..fH.~.fH.:...
0x7f92d08e48ef 48 09 c6 48 09 d6 41 0f b6 c2 41 0f b6 53 09 48 H..H..A...A..S.H

Native stacktrace:

/usr/bin/mono(+0x129f8d) [0x5585b949cf8d]
/usr/bin/mono(+0x12a295) [0x5585b949d295]
/usr/bin/mono(+0xbfd2f) [0x5585b9432d2f]
/usr/bin/mono(+0x414e8) [0x5585b93b44e8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f92d3d3e890]
/usr/lib/mono/aot-cache/amd64/mscorlib.dll.so(+0x1238cf) [0x7f92d08e48cf]
@trolley

This comment has been minimized.

Copy link

@trolley trolley commented Dec 24, 2018

I have the same result as Thf772.

@noabody

This comment has been minimized.

Copy link

@noabody noabody commented Dec 25, 2018

Ubuntu 18.04:
I have multiple computers and all updated except older Athlon 9850BE which is notable because it doesn't have SSE 4.1, only 4a. Since all six computers with SSE4 updated cleanly and the two without, failed, I wondered if mono now requires that instruction set.

And yes, I do have 8 computers running Ubuntu 18.04.

@cdanderson

This comment has been minimized.

Copy link

@cdanderson cdanderson commented Dec 28, 2018

If you purge manually it will work.

Is this what you meant @simeon4110?

sudo dpkg --remove --force-remove-reinstreq ca-certificates-mono

Unfortunately that didn't work and I received the following error:

dpkg: dependency problems prevent removal of ca-certificates-mono:
 mono-devel depends on ca-certificates-mono (= 5.18.0.225-0xamarin1+ubuntu1804b1); however:
  Package ca-certificates-mono is to be removed.

dpkg: error processing package ca-certificates-mono (--remove):
 dependency problems - not removing
Errors were encountered while processing:
 ca-certificates-mono

Can I ask you to please be more specific with how this can be manually purged? Thank you!

@lightway82

This comment has been minimized.

Copy link

@lightway82 lightway82 commented Dec 28, 2018

I decided this in a couple of hours by reinstalling the system. It turned out faster and easier.
After that, I build the library from source and use it without installation.

@noabody

This comment has been minimized.

Copy link

@noabody noabody commented Dec 28, 2018

The noted regex work on Ubuntu 18.04 and I recommend not removing the PPA from software lists because that will hinder some of these. Once removal is satisfactorily complete, then remove the PPA.

Identify installed packages (dpkg):
echo $(dpkg -l '*mono*' | grep -Pio '(?<=^ii  )(mono|libmono)-.+?(?=\s.*)' | sort)

Simulate purge:
apt -s purge $(dpkg -l '*mono*' | grep -Pio '(?<=^ii  )(mono|libmono)-.+?(?=\s.*)' | sort)

Purge orphaned configs:
sudo dpkg --purge $(dpkg -l '*mono*' | grep -Pio '(?<=^rc  ).+?(?=\s.*)')

Identify PPA  and pre-format for command (ppa-purge):
find /var/lib/apt/lists -maxdepth 1 -type f -iname '*_*packages' -printf '%P\n' | perl -pe 's/(.+?)_(.+?)_(.+?)_.*/-s \1 -o \2 -p \3/gi' | grep -Pio '^((?!dists).)*$' | sort | uniq

Identify PPA packages from package list (definitive):
echo $(find /var/lib/apt/lists -maxdepth 1 -type f -iname '*mono-project*_*packages' -exec grep -Pio '(?<=^package: ).*' "{}" \; | sort | uniq)

Identify PPA installed packages:
echo $(apt -s purge $(find /var/lib/apt/lists -maxdepth 1 -type f -iname '*mono-project*_*packages' -exec grep -Pio '(?<=^package: ).*' "{}" \;) | grep -Pio '(?<=^purg ).+?\s' | sort | uniq)

Hold PPA installed packages:
sudo apt-mark hold $(apt -s purge $(find /var/lib/apt/lists -maxdepth 1 -type f -iname '*winehq*_*packages' -exec grep -Pio '(?<=^package: ).*' "{}" \;) | grep -Pio '(?<=^purg ).+?\s' | sort | uniq)

The grep might be helpful to identify core packages like these:

mono-gac mono4.0-gac mono-devel mono-complete mono-dbg ca-certificates-mono mono-xsp4 libmono-addins-msbuild-cil-dev libmono-addins-gui-cil-dev libmono-addins-cil-dev libmono-reflection-cil  libmono-addins-msbuild0.2-cil libmono-zeroconf-cil-dev

sudo apt removing the core will cascade removal of dependent packages then it's a matter of installing stock mono. Checking dpkg for leftover configs could help:

/usr/share/cli-common/policies.d/
libglade2.0-cil libglib2.0-cil libgtk2.0-cil libnunit-console-runner2.6.3-cil libnunit-core2.6.3-cil libnunit-core-interfaces2.6.3-cil libnunit-framework2.6.3-cil libnunit-mocks2.6.3-cil libnunit-util2.6.3-cil

/usr/lib/cli/
glade-sharp-2.0 glib-sharp-2.0 gtk-sharp-2.0 Mono.Addins-0.2 Mono.Addins.Gui-0.2 nunit-console-runner-2.6.3 nunit.core-2.6.3 nunit.core.interfaces-2.6.3 nunit.framework-2.6.3 nunit.mocks-2.6.3 nunit.util-2.6.3

(leave alone unless complete purge)
/usr/lib/mono/gac
/etc/mono

There were crash-logs in the folders mentioned above that should be easy to purge manually with a find command.

@simeon4110

This comment has been minimized.

Copy link

@simeon4110 simeon4110 commented Dec 29, 2018

@noabody

This comment has been minimized.

Copy link

@noabody noabody commented Dec 29, 2018

A script would be nice. Thing is, we could use ppa-purge to some extent. Have to dig into the script to see how it orders options and it doesn't cleanly remove the PPA entries in /etc/apt/sources.list

The bigger issue is that mono compiles certain components upon install. That's where my non-SSE4 processor failed to create the necessary 'exe' that in turn failed to complete the installation, leaving it broken.

Once that happens, there'll be warnings about non-empty folders (/usr/share/cli-common/policies.d /usr/lib/cli /usr/lib/mono/gac /etc/mono) and, upon purge, the user will not know if it is safe to remove them.

Sampling of commands to both install ppa and purge:

sudo apt-add-repository 'deb https://download.mono-project.com/repo/ubuntu stable-bionic main'
echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo ppa-purge -s download.mono-project.com -o repo -p ubuntu
@Nabile-Rahmani

This comment has been minimized.

Copy link

@Nabile-Rahmani Nabile-Rahmani commented Dec 30, 2018

@noabody I seriously hope this new SSE requirement is a mistake because I just updated Mono on my 2011 Debian 9 system and I'm also getting these SIGILLs.
I'm using it as a build server so that sucks. Heck, I even dev on it from time to time. Surely that's not old enough hardware to get dropped, right ? Aren't there fallbacks for compatibility ?

I'm also not gonna tempt updating and possibly breaking Mono on my main (more recent) development machine until this is figured out :s

@Nichtraucher

This comment has been minimized.

Copy link
Author

@Nichtraucher Nichtraucher commented Jan 1, 2019

Perhaps a developer can chime in and clear this up, please?

@simeon4110

This comment has been minimized.

Copy link

@simeon4110 simeon4110 commented Jan 1, 2019

@noabody

This script will purge mono and install a stable version. I have tested it with Ubuntu 18.04, but it should work with anything after 16.04. It may seem like it gets caught in an escape loop, just let it keep running until it gets the job done.

USE AT YOUR OWN RISK: There is no error checking done with this script.

You will need to install any non devel packages once it finishes.

#!/bin/bash

# Function removes all old versions of mono, swaps out mono release repo for a 
# stable repo, and installs the stable version.
revert_to_old_mono() {
  apt remove mono* libmono-*

  # Change apt repo URL.
  mv /etc/apt/sources.list.d/mono-official.list /etc/apt/sources.list.d/mono-offical.list.old
  touch /etc/apt/sources.list.d/mono-official.list
  echo "deb http://download.mono-project.com/repo/ubuntu bionic/snapshots/5.16.0.220 main" | tee /etc/apt/sources.list.d/mono-official.list

  # Chown and chmod the new file.
  chown root:root /etc/apt/sources.list.d/mono-official.list
  chmod 0644 /etc/apt/sources.list.d/mono-official.list

  # Update and install mono version.
  apt update
  apt install mono-devel -y
}

# Reverses changes from revert_to_old_mono.
revert_to_main_repo_mono() {
  rm /etc/apt/sources.list.d/mono-official.list
  mv /etc/apt/sources.list.d/mono-offical.list.old /etc/apt/sources.list.d/mono-official.list

  apt update && apt upgrade -y
}

#### main
case "$1" in
  -f|--run-fix )
  revert_to_old_mono ;;
  -r|--revert-changes )
  revert_to_main_repo_mono ;;
  * )
  echo ""
  echo "Enter -f to run the fix or -r to revert the changes."
  echo "" ;;
esac

To run the fix, run this script with the '-f' flag.

Once the devs write a patch (I'd offer to help if I could write C#) run the script with the '-r' flag to roll back the changes to apt.

@marek-safar

This comment has been minimized.

Copy link
Member

@marek-safar marek-safar commented Jan 2, 2019

@directhex could you look into this?

@directhex

This comment has been minimized.

Copy link
Contributor

@directhex directhex commented Jan 2, 2019

It looks like @noabody figured it out - we hardcode SSE4.1 in LLVM AOT:

mono/mono/mini/aot-compiler.c

Lines 1091 to 1099 in 40ec59d

#if defined(TARGET_X86)
g_string_append (acfg->llc_args, " -march=x86 -mattr=sse4.1");
#endif
#if defined(TARGET_AMD64)
g_string_append (acfg->llc_args, " -march=x86-64 -mattr=sse4.1");
/* NOP */
acfg->align_pad_value = 0x90;
#endif

This can be worked around much more easily by not installing the package mono-llvm-support (or removing it if already installed, with dpkg --purge mono-llvm-support)

@marek-safar We should probably discuss the right approach to take here - should that hardcoded instruction set assumption remain? Should LLVM AOT remain the default (over classic AOT)? Bear in mind I can make changes to the Linux packages which are distinct from changes to the main mono.git

The problem has likely existed for a few versions, going by the git commits, but due to a packaging bug I haven't bothered to fix, the first time Mono is installed, Corlib isn't LLVM AOT'd - it's an issue on upgrade or reinstall, so it's been masked for a long time.

@directhex

This comment has been minimized.

Copy link
Contributor

@directhex directhex commented Jan 2, 2019

OK, I've rewritten the postinst script to specifically check for SSE4.1, and always disable LLVM AOT if it's not available. I'm doing an emergency build with the fix into the vs repo of 5.16.0.220, and into preview of 5.18.0.225 - if/when that succeeds I'll promote the 5.18 build into the stable branch

@directhex

This comment has been minimized.

Copy link
Contributor

@directhex directhex commented Jan 2, 2019

Postinst also now gives more detail on its behaviour - i.e. it'll say if it's trying with LLVM to give users more clarity. Once the fixed packages land, users will be able to resolve the problem with an update.

@pracplayopen

This comment has been minimized.

Copy link

@pracplayopen pracplayopen commented Jan 3, 2019

having same issue on ubuntu 16.04 systems. could make sense if llvm update was involved, successfully removing mono-llvm doesn't fix that both mono processes and mono updates fail w/the mono_crash dump loop others experiencing above..

like other users here, trying to purge the mono-certificates package failed. never had an issue of this severity before with non-edge update.

not 100% sure what changes to make w/regard to running simeon4110 's 18.04 uninstall script, suppose will look into that.

@pracplayopen

This comment has been minimized.

Copy link

@pracplayopen pracplayopen commented Jan 3, 2019

ok had to run it multiple times to get the uninstall to complete (because of so many dumps), but this has modifications to @simeon4110 to restore mono (and could work w/other dists if you change value of dist). thx

#!/bin/bash
#dist=bionic/snapshots/5.16.0.220
#dist="stable-xenial"
dist="stable-xenial/snapshots/5.16.0.220"
#distlistpath=/etc/apt/sources.list.d/mono-official.list
distlistpath=/etc/apt/sources.list.d/mono-official-stable.list 
# Function removes all old versions of mono, swaps out mono release repo for a 
# stable repo, and installs the stable version.
revert_to_old_mono() {
  apt remove mono* libmono-*

  # Change apt repo URL.
  mv $distlistpath "$distlistpath.old"
  touch $distlistpath
  echo "deb http://download.mono-project.com/repo/ubuntu $dist main" | tee $distlistpath

  # Chown and chmod the new file.
  chown root:root $distlistpath
  chmod 0644 $distlistpath

  # Update and install mono version.
  apt update
  apt install mono-devel -y
}

# Reverses changes from revert_to_old_mono.
revert_to_main_repo_mono() {
  rm $distlistpath
  mv "$distlistpath.old" $distlistpath

  apt update && apt upgrade -y
}

#### main
case "$1" in
  -f|--run-fix )
  revert_to_old_mono ;;
  -r|--revert-changes )
  revert_to_main_repo_mono ;;
  * )
  echo ""
  echo "Enter -f to run the fix or -r to revert the changes."
  echo "" ;;
esac
@ghost

This comment has been minimized.

Copy link

@ghost ghost commented Jan 3, 2019

I experienced the same issue when updating to 5.18. I was still able to update to it though.

The update failed but, just to see what would happen I then tried a complete removal (via a combination of aptitude and dpkg) and a full clean install to 5.18. Which did take.

However, something did break and has remained broken ever since -- I can no longer compile anything. For the very same reason as the OP's report. Errors in either the runtime or in native code. In this case, it's a SIGILL. And this is the case for any project, any amount of code. No matter how simple, it will not compile.

I suspect it's the very same problem deep down because the update that originally failed failed on the precompile step of some of the components.

Just thought I'd mention that it's not just updating that is failing. Using 5.18 is also less than worry free.

@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Jan 3, 2019

Another workaround is to export MONO_ENV_OPTIONS=-O=-aot. That tells the runtime to always avoid AOT'd code and fall back to the JIT.

@ghost

This comment has been minimized.

Copy link

@ghost ghost commented Jan 3, 2019

Can confirm that does work indeed. So, that does suggest we've found the culprit indeed. Good to know. Looking forward to seeing that remedied forthwith with a patch.

@noabody

This comment has been minimized.

Copy link

@noabody noabody commented Jan 3, 2019

Intel® Software Development Emulator Download

Deailed information:

Apparently emulates SSE 4.1. As it was when SSE2/SSE3 became requirements for much of the software in use today, so too will it be with SSE4.1. Those of us with older, usable machines, know how this plays out.

@ghost

This comment has been minimized.

Copy link

@ghost ghost commented Jan 3, 2019

Emulating SSE4.1 is no solution though, if I may be so bold.

  • It is not going to be adopted widespread, emulation of SSE4.1. Can't very well expect people to install external tools just to continue using the one that broke in the first place. That's just messy.
  • Emulating it is probably slower than not using it at all. OK, gross assumption but probably not untrue.
  • Implementing a conditional in Mono is cleaner. Makes me wonder why one did not exist before release of 5.18.

For the time being, @lewurm's workaround of simply disabling the AOT optimization is the cleanest workaround (removing the mono llvm package can and probably will lead to dependency issues) until a conditional is implemented, to check for SSE4.1 support in the hardware.

@directhex

This comment has been minimized.

Copy link
Contributor

@directhex directhex commented Jan 3, 2019

Implementing a conditional in Mono is cleaner. Makes me wonder why one did not exist before release of 5.18.

Can't the CPU capabilities be probed via CPUID and the LLVM SSE4.1 optimisation added conditionally with said capability listing ?

Probably! It just... wasn't done that way.

What we're doing is just passing command-line arguments to llc, and unhelpfully, llc doesn't have anything like gcc's -march=native to optimize for the detected CPU.

@Nabile-Rahmani

This comment has been minimized.

Copy link

@Nabile-Rahmani Nabile-Rahmani commented Jan 3, 2019

mattr, along with mcpu seem like they're defaulting to optimising for the automatically detected CPU, but I have no idea if it actually does this (or if it's even in the version in use by Mono).

But then again, the defaults for the chip might not use all the chip's features in the first place ? Who knows, though that'd be a weird choice 🤔

@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Mar 11, 2019

let's keep it open until we have propagated the fix to release branches.

@directhex

This comment has been minimized.

Copy link
Contributor

@directhex directhex commented Mar 21, 2019

Before making changes which affect everyone (given how badly it went last time), I'd like to enlist help from community people, especially with older CPUs, to ensure the default works as expected.

@sbrl

This comment has been minimized.

Copy link

@sbrl sbrl commented Mar 21, 2019

@directhex: Given an archive (e.g. .tar.gz, .zip) containing a compiled binary to test, I can do so on the following CPUs that do not have SSE4.1:

Raspberry Pi 3B+

Architecture:          armv7l
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
Model:                 4
Model name:            ARMv7 Processor rev 4 (v7l)
CPU max MHz:           1200.0000
CPU min MHz:           600.0000
BogoMIPS:              38.40
Flags:                 half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
Intel Atom N2800

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  2
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               54
Model name:          Intel(R) Atom(TM) CPU N2800   @ 1.86GHz
Stepping:            1
CPU MHz:             799.833
CPU max MHz:         1862.0000
CPU min MHz:         798.0000
BogoMIPS:            3732.99
L1d cache:           24K
L1i cache:           32K
L2 cache:            512K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts nopl nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm dtherm arat
@lateralusX lateralusX closed this Mar 26, 2019
@lewurm lewurm reopened this Mar 26, 2019
@jaykrell

This comment has been minimized.

Copy link
Collaborator

@jaykrell jaykrell commented Mar 26, 2019

There's no great solution to any of this.
People do various things.

  • compile .dll/.so multiple times and have runtime loader pick best, possibly fallback to JIT if none of the images work on current machine
  • compile some functions multiple times and make a runtime choice -- but which functions?
  • target low common denominators, giving up perf on most machines - if filesystem was more cloud-based, then targeting several processors and probing would be more practical
@jaykrell

This comment has been minimized.

Copy link
Collaborator

@jaykrell jaykrell commented Apr 24, 2019

-march=native has major downside => “it works on my machine”

@marek-safar marek-safar removed this from the 2019-02 (6.0.xx) milestone May 30, 2019
@Nabile-Rahmani

This comment has been minimized.

Copy link

@Nabile-Rahmani Nabile-Rahmani commented Oct 21, 2019

@jaykrell Well in this case the Mono installation script targets the user's machine, so it has to generate code optimised and supported for that specific machine only. Hence relying on LLVM auto-detection.
Seeing as the fallback AOT has worked fine on all hardware since forever, I am inclined to think SSE4.1 isn't actually a requirement to generate AOT code with LLVM.

After waiting for GitHub to display blames, I ended up looking at the commit that introduced the requirement: 628aa55#diff-499ab2170cdda019ebc31342389233b3L670

Could @vargaz clarify if this flag was added just for testing purposes or if it was vital for LLVM to even work ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.