Callbacks not firing?

[justcfx2u]

I'm getting a very similar error same as this. https://forum.fivem.net/t/fxserver-random-error-s/37333

It seems to be related to using MySQL.
Trying to terminate the TCP connection to MySQL at runtime does not seem to cause this error (using tcpkill, iptables, etc).

We do not necessarily get the following error, but we have had it before:

System.ArgumentException: Destination array was not long enough. Check destIndex and length, and the array's lower bounds
  at System.Array.Copy (System.Array sourceArray, System.Int32 sourceIndex, System.Array destinationArray, System.Int32 destinationIndex, System.Int32 length) [0x000c1] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at System.Collections.Generic.List`1[T].ToArray () [0x0000c] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x00000] in /src/code/client/clrcore/CitizenTaskScheduler.cs:96
  at CitizenFX.Core.InternalManager.TriggerEvent (System.String eventName, System.Byte[] argsSerialized, System.String sourceString) [0x000ff] in /src/code/client/clrcore/InternalManager.cs:210

This error sometimes happens ONCE during the course of server operation, and the server continues to function...

Error during Tick: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Tasks.TaskScheduler.TryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.InvokeTryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in /src/code/client/clrcore/CitizenTaskScheduler.cs:130
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x00029] in /src/code/client/clrcore/CitizenTaskScheduler.cs:107
  at CitizenFX.Core.InternalManager.Tick () [0x00076] in /src/code/client/clrcore/InternalManager.cs:174

We have also gotten a variant:

System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Tasks.TaskScheduler.TryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.InvokeTryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in /src/code/client/clrcore/CitizenTaskScheduler.cs:130
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x00029] in /src/code/client/clrcore/CitizenTaskScheduler.cs:107
  at CitizenFX.Core.InternalManager.TriggerEvent (System.String eventName, System.Byte[] argsSerialized, System.String sourceString) [0x000ff] in /src/code/client/clrcore/InternalManager.cs:210

(note in this case there is no "Error during Tick")

But when this begins happening...

Error during Tick: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Tasks.TaskScheduler.TryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.InvokeTryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in /src/code/client/clrcore/CitizenTaskScheduler.cs:130
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x00029] in /src/code/client/clrcore/CitizenTaskScheduler.cs:107
  at CitizenFX.Core.InternalManager.Tick () [0x00076] in /src/code/client/clrcore/InternalManager.cs:174
Error during Tick: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Tasks.TaskScheduler.TryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.InvokeTryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in /src/code/client/clrcore/CitizenTaskScheduler.cs:130
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x00029] in /src/code/client/clrcore/CitizenTaskScheduler.cs:107
  at CitizenFX.Core.InternalManager.Tick () [0x00076] in /src/code/client/clrcore/InternalManager.cs:174
Error during Tick: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Tasks.TaskScheduler.TryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.CitizenTaskScheduler.InvokeTryExecuteTask (System.Threading.Tasks.Task task) [0x00000] in /src/code/client/clrcore/CitizenTaskScheduler.cs:130
  at CitizenFX.Core.CitizenTaskScheduler.Tick () [0x00029] in /src/code/client/clrcore/CitizenTaskScheduler.cs:107
  at CitizenFX.Core.InternalManager.Tick () [0x00076] in /src/code/client/clrcore/InternalManager.cs:174

as quickly as the server can output the messages, the game continues to run (and scripts generally seem to execute) but all database queries do not fire.

We've had Error during Tick: System.NullReferenceException occur after a SELECT COUNT(*) (fetchScalar), SELECT (fetchScalar), and UPDATE (execute). It does not seem to happen when using fetchAll() during testing, but it may exist.

I have found no correlation of query type (SELECT, UPDATE) or single statement which causes this.

The error seems to happen after several minutes of server uptime, and is fairly reproducible using code similar to the following:

function selectCharacters()                                                                                                          [1/9274]
  MySQL.Async.fetchAll('SELECT * FROM characters LIMIT 1', {}, function(allChars)
    print('select ' .. #allChars)
  end)
  SetTimeout(1, selectCharacters)
end

function updateBlah()
  MySQL.Async.execute("UPDATE vtable SET Blah=100 LIMIT 1", {}, function(modifiedRows)
    print('blah modified ' .. modifiedRows)
  end)
  SetTimeout(1, updateBlah)
end

AddEventHandler('onMySQLReady', function()
  SetTimeout(1, selectCharacters)
  SetTimeout(1, updateBlah)
end)

This has occurred with artifacts 284 and 294 (and likely others) of FiveM proot on Linux.

https://runtime.fivem.net/artifacts/fivem/build_proot_linux/master/294-b2423af33c0033b34d47f1d6ec42a49d12127efa/

We have tried using the stock clone of this repository's master (1a98024), and we have also tried by building MySqlConnector v0.23.0 (mysql-net/MySqlConnector) on Linux using mono. After that, we have also tried replacing the .dll's that come with mysql-async with .dll's from the runtime:

/usr/share/dotnet/shared/Microsoft.NETCore.App/1.1.2/System.Runtime.InteropServices.RuntimeInformation.dll
/usr/share/dotnet/shared/Microsoft.NETCore.App/1.1.2/System.Threading.Tasks.Extensions.dll
/usr/share/dotnet/shared/Microsoft.NETCore.App/1.1.2/System.Buffers.dll

After copying the .dll's from the runtime it seems to run a little longer, but it still ultimately enters the "Error on Tick" loop which prevents any queries from completing.

[justcfx2u]

I also tried using a Sync version. I believe this is the same thing, but a different perspective.
Our MySQL server is on a remote host.

function selectCharacters()
  local allChars = MySQL.Sync.fetchAll('SELECT * FROM characters LIMIT 1')
  print('select ' .. #allChars)
  SetTimeout(1, selectCharacters)
end

function updateFuel()
  local modifiedRows = MySQL.Sync.execute("UPDATE vtable SET Blah=100 LIMIT 1")
  print('Blah modified ' .. modifiedRows)
  SetTimeout(1, updateFuel)
end

AddEventHandler('onMySQLReady', function()
  SetTimeout(1, selectCharacters)
  SetTimeout(1, updateFuel)
end)

An error happens when opening MySQL connection Connect Timeout expired.
[0ms] An error happens when executing SELECT * FROM characters LIMIT 1 : Connection must be Open; current state is Closed
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException:
 Connection must be Open; current state is Closed
  at MySqlAsync.MySqlAsync.ExecuteReader (System.String query, System.Collections.Generic.IDictionary`2[TKey,TValue] parameters) [0x00053] in
 <5a6a1f5752e64a84b656c628ba674f61>:0
  at MySqlAsync.MySqlAsync.<MySqlAsync>m__4 (System.String query, System.Collections.Generic.IDictionary`2[TKey,TValue] parameters) [0x00000]
 in <5a6a1f5752e64a84b656c628ba674f61>:0
  at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, Syst
em.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
   --- End of inner exception stack trace ---

Server stack trace:                                                                                                     
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, Syst
em.Object[] parameters, System.Globalization.CultureInfo culture) [0x00048] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at System.Delegate.DynamicInvokeImpl (System.Object[] args) [0x000e7] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at System.MulticastDelegate.DynamicInvokeImpl (System.Object[] args) [0x00008] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at System.Delegate.DynamicInvoke (System.Object[] args) [0x00000] in <d7e663f2f7cd4ab6929018ec5233f09d>:0
  at CitizenFX.Core.FunctionReference.Invoke (System.Int32 reference, System.Byte[] arguments) [0x0004f] in /src/code/client/clrcore/Function
Reference.cs:62
  at CitizenFX.Core.InternalManager.CallRef (System.Int32 refIndex, System.Byte[] argsSerialized, System.IntPtr& retvalSerialized, System.Int
32& retvalSize) [0x00000] in /src/code/client/clrcore/InternalManager.cs:235
  at (wrapper remoting-invoke-with-check) CitizenFX.Core.InternalManager:CallRef (int,byte[],intptr&,int&)
  at (wrapper xdomain-dispatch) CitizenFX.Core.InternalManager:CallRef (object,byte[]&,byte[]&,int,byte[],int&)

Exception rethrown at [0]:
  at (wrapper xdomain-invoke) CitizenFX.Core.InternalManager:CallRef (int,byte[],intptr&,int&)
  at (wrapper remoting-invoke-with-check) CitizenFX.Core.InternalManager:CallRef (int,byte[],intptr&,int&)
  at CitizenFX.Core.MonoScriptRuntime.CallRef (System.Int32 refIndex, System.Byte[] argsSerialized, System.Int32 argsSize, System.IntPtr& ret
valSerialized, System.Int32& retvalSize) [0x00013] in /src/code/client/clrcore/MonoScriptRuntime.cs:151
Error resuming coroutine: citizen:/scripting/lua/MessagePack.lua:830: missing bytes
stack traceback:
        [C]: in function 'error'
        citizen:/scripting/lua/MessagePack.lua:830: in method 'underflow'
        citizen:/scripting/lua/MessagePack.lua:465: in field 'any'
        citizen:/scripting/lua/MessagePack.lua:860: in field 'unpack'
        citizen:/scripting/lua/scheduler.lua:338: in function <citizen:/scripting/lua/scheduler.lua:331>
        (...tail calls...)
        explode.lua:3: in function 'selectCharacters'
^C^C

But in this case, the clients timed out and the server seems to have stopped running completely.

Edit: It seems like I was able to mitigate this and get a longer-running program by adding ConnectionTimeout=120;keepalive=0;pooling=true to the Sync versions. The Async versions still fails within 15 minutes with ConnectionTimeout, keepalive, and pooling enabled.

[justcfx2u]

Looks like this was solved on the FiveM side, beginning with artifact 298 and later. This issue can be closed.

[joelwurtz]

Awesome repro by the way, will leave it open to see if no other people have callback issue, i have currently more time to look at this.

[joelwurtz]

Should be good, did not have this issue anymore