update to 5.18.0.268-0xamarin1+ubuntu1404b1 fails

[Nichtraucher]

Steps to Reproduce

1. update to 5.18.0.225-0xamarin1+ubuntu1404b1

Current Behavior

update fails due to unresolved dependencies and other errors (see terminal output below)

Expected Behavior

the update goes through smoothly

On which platforms did you notice this

[ ] macOS
[x ] Linux
[ ] Windows

Version Used:

Linux version 4.4.0-142-generic (buildd@lcy01-amd64-006) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu114.04.4) # 16814.04.1-Ubuntu SMP Sat Jan 19 11:26:28 UTC 2019
I'm using the repositories provided by mono.

Mono version:

Mono JIT compiler version 5.18.0.268 (tarball Thu Feb 28 15:28:45 UTC 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
Interpreter: yes
LLVM: yes(600)
Suspend: preemptive
GC: sgen (concurrent by default)

Stacktrace

terminal output here: https://pastebin.com/2wVtmGDQ

I assume that this is the same or a similar issue as the one that I've reported previously here: #12182 ?

[marek-safar]

/cc @directhex

[lewurm]

#3  0x000000000042fc98 in ?? ()
#4  <signal handler called>
#5  0x00007fccff4e535f in uint16_ToString_string_System_IFormatProvider () from /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
#6  0x00007fccff4e52c0 in ?? () from /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
#7  0x0000000000f47ca0 in ?? ()
#8  0x0000000000000000 in ?? ()

This is a crash in AOT'd code. Could be a bug triggered by a different locale (German in this case).

@Nichtraucher could you post the output of env on the command line? Also, as a workaround, could you try to run the apt command with MONO_ENV_OPTIONS=-O=-aot apt install [...]? This will disable the usage of AOT'd code and fallback to the just-in-time compiler.

[Nichtraucher]

sure

~ $ env

XDG_VTNR=8
MATE_DESKTOP_SESSION_ID=this-is-deprecated
SSH_AGENT_PID=2083
XDG_SESSION_ID=c1
GPG_AGENT_INFO=/run/user/1000/keyring-u7ewRG/gpg:0:1
TERM=xterm
SHELL=/bin/bash
XDG_SESSION_COOKIE=e283802f0664321f988a3659596f9a2c-1551432200.952533-110002306
DERBY_HOME=/usr/lib/jvm/java-8-oracle/db
WINDOWID=69206045
GNOME_KEYRING_CONTROL=/run/user/1000/keyring-u7ewRG
USER=benni
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
SSH_AUTH_SOCK=/run/user/1000/keyring-u7ewRG/ssh
SESSION_MANAGER=local/benni-M2NS-NVM:@/tmp/.ICE-unix/1859,unix/benni-M2NS-NVM:/tmp/.ICE-unix/1859
USERNAME=benni
DEFAULTS_PATH=/usr/share/gconf/mate.default.path
XDG_CONFIG_DIRS=/etc/xdg/xdg-mate:/etc/xdg
PATH=/home/benni/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin
DESKTOP_SESSION=mate
GDM_XSERVER_LOCATION=local
PWD=/home/benni
JAVA_HOME=/usr/lib/jvm/java-8-oracle
GNOME_KEYRING_PID=2175
LANG=de_DE.UTF-8
MANDATORY_PATH=/usr/share/gconf/mate.mandatory.path
MDM_XSERVER_LOCATION=local
GDMSESSION=mate
SHLVL=1
XDG_SEAT=seat0
HOME=/home/benni
XDG_SESSION_DESKTOP=mate
LOGNAME=benni
XDG_DATA_DIRS=/usr/share/mate:/usr/local/share/:/usr/share/:/usr/share/mdm/
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-qJombMODTG,guid=8c131e905e685b028b04aa635c78fa0f
J2SDKDIR=/usr/lib/jvm/java-8-oracle
LESSOPEN=| /usr/bin/lesspipe %s
MDMSESSION=mate
TEXTDOMAIN=im-config
WINDOWPATH=8
XDG_RUNTIME_DIR=/run/user/1000
DISPLAY=:0.0
MDM_LANG=de_DE.UTF-8
XDG_CURRENT_DESKTOP=MATE
J2REDIR=/usr/lib/jvm/java-8-oracle/jre
LESSCLOSE=/usr/bin/lesspipe %s %s
TEXTDOMAINDIR=/usr/share/locale/
COLORTERM=mate-terminal
XAUTHORITY=/home/benni/.Xauthority
_=/usr/bin/env

~ $ MONO_ENV_OPTIONS=-O=-aot apt install [...] &> output.txt

buttlong output here: https://pastebin.com/KL4fMWhd

[directhex]

What is your CPU?

Can you delete /usr/lib/mono/aot-cache/amd64 and try what you were doing again?

[Nichtraucher]

What is your CPU?

~ $     cat /proc/cpuinfo
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 107
model name	: AMD Processor model unknown
stepping	: 2
cpu MHz		: 1800.000
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good nopl pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch vmmcall lbrv
bugs		: apic_c1e fxsave_leak sysret_ss_attrs spectre_v1 spectre_v2
bogomips	: 3616.90
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 107
model name	: AMD Processor model unknown
stepping	: 2
cpu MHz		: 1800.000
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good nopl pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch vmmcall lbrv
bugs		: apic_c1e fxsave_leak sysret_ss_attrs spectre_v1 spectre_v2
bogomips	: 3616.90
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps

Can you delete /usr/lib/mono/aot-cache/amd64 and try what you were doing again?

That helped - the update went through.

[noabody]

Update failed for me on AMD K7, K10 and Intel Pineview. I used these steps on Ubuntu 18.04.2:

cd /usr/lib/mono/
sudo rm -rf aot-cache
var="MONO_ENV_OPTIONS" ; val="-O=-aot" ; (grep -Pi "$var" "$HOME/.profile") && perl -pi -e "s|(.*$var)=.*|\\1=$val|gi" "$HOME/.profile" || echo "export $var=$val" >> "$HOME/.profile"

Logout then back in to get the .profile environment variable to set. Presumably, these older non-sse4.1 machines will only work with the JIT AOT as opposed to the newer implementation.

I do a quick check on .profile to populate the MONO_ENV_OPTIONS if it doesn't exist or update the value if it does.

[directhex]

As per #12182, LLVM AOT is only used on systems with SSSE4.1 - did we change our optimizations for legacy AOT?

[nSpecter]

If it helps any, I'm having the exact same issue on an AMD Athlon LE-1640 single core, 2698.91Mhz CPU on Lubuntu 18.04.2 LTS.

Also had the EXACT same issue on the December 24th, 2018 Mono update which took me about 6 hours to stumble through a fix. Still not certain what the ultimate solution was, but many hours were spent figuring out how to uninstall Mono components that refused to uninstall, followed by re-installing components. ¯\_(ツ)_/¯

I'll try the fixes listed above and report back.

---

Edit:

Removing the amd64 directory with its attendant mscorelib.dll.so file did produce a change. Upon running sudo apt --fix-broken install it advanced further, ending with:

...
Setting up libmono-cil-dev (5.18.0.268-0xamarin7+ubuntu1804b1) ...
Setting up mono-devel (5.18.0.268-0xamarin7+ubuntu1804b1) ...
Setting up mono-complete (5.18.0.268-0xamarin7+ubuntu1804b1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
W: APT had planned for dpkg to do more than it reported back (377 vs 381).
Affected packages: mono-gac:amd64

I then ran, sudo apt-get upgrade

The following packages will be upgraded:
gir1.2-dbusmenu-glib-0.4 kodi kodi-x11 libdbusmenu-glib4 libdbusmenu-gtk3-4
libdbusmenu-gtk4 libgd3 libmono-btls-interface4.0-cil libnss3 libssl1.0.0
10 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 32.8 MB/35.3 MB of archives.
After this operation, 1,024 B disk space will be freed.

...which completed without errors.

Now, for a reboot....

---

Edit:

Reboot was a success and all three of my Mono-based daemons appear to be up and running fine.

I wish this post was around last Christmas. It's a much more elegant solution than what I came up with. :)

[noabody]

Ubuntu users might want to clear /var/crash to avoid being spammed by "A system problem has been detected" messages.

ls -l /var/crash/
total 1528
-rw------- 1 root whoopsie  365988 Mar  1 06:34 mono-gac.0.crash
-rw-r----- 1 root whoopsie 1194871 Mar  1 06:32 _usr_lib_mono_4.5_gacutil.exe.0.crash
sudo rm /var/crash/*

I have to say that this particular update failure is truly terrifying so when I read:

Also, as a workaround, could you try to run the apt command with MONO_ENV_OPTIONS=-O=-aot apt install [...]? This will disable the usage of AOT'd code and fallback to the just-in-time compiler.

Then:

"Can you delete /usr/lib/mono/aot-cache/amd64 and try what you were doing again?"

There was no way I was going to take a chance on having the problem reoccur. Better to set the environment variable in .profile, to ensure SSE4.1 optimized AOT doesn't have the slightest chance of slipping through. I know which of my computers are affected.

It just isn't worth the headache.

[directhex]

You shouldn't get the SSSE4.1-requiring AOT on CPUs which don't support it - it's specific to LLVM AOT, which is disabled on CPUs which don't support it.

This bug seems more like we made changes to our optimizations in non-LLVM AOT? I'll have to investigate the AOT code

[directhex]

I can't verify that you're definitely not producing LLVM AOT, without a dpkg-reconfigure libmono-corlib4.5-cil output