Possible memory leak #1223
Comments
|
After running the application for 13 hours this crash is ocurring. |
|
|
|
The error in the previous screenshot seems to be out of memory? |
|
@tmikov yes, also the cpu goes all the way up when the crash begins. |
|
Everybody is on a break now, but we will take a more serious look after the holidays. |
|
@tmikov thank you for replying and happy holidays! |
|
Any updates on this stacktrace? |
|
Hey @lucaswitch, we'd need more information, like a minimal repro to offer more insight here. It looks like you may be running into two separate problems, the first is a crash somewhere in the GC, and the second is an OOM. The OOM is likely caused by a large number of allocations, and you can try debugging it by connecting to the application in DevTools and taking a heap snapshot or an allocation timeline profile. That should help you pin down where the allocations are coming from. It is also worth monitoring the runtime's reporting memory consumption by calling The crash is trickier and may indicate a bug in Hermes, unless Hermes is being used incorrectly (e.g. from multiple threads simultaneously). Are you able to share a minimised JS repro that results in the crash? |
|
Thank you for replying, I'm also debugging for possible causes that could create the OOM or any kind of over memory/cpu usage. |
|
@neildhar Does this getInstrumentedStats looks normal: js_totalAllocatedBytes looks too big but not sure if it's to be that way on debug mode. Is there a documentation about the getInstrumentedStats attributes meaning? |
|
32MB heap, looks totally normal. |
|
So i think the oom is solved(we did a full refactor of the app), but the crash sadly persists the same. Also now i'm keeping eye on the getInstrumentedStats all the time and seems normal(only js_totalAllocatedBytes gets quite big) I will consider that some module could be messing up the runtime for now. I'm looking some variables at the moment of the crash to indicate which module could be causing this crash then i can take a deeper look at It. And then make a repro(or remove it). Looking after those stacktrace i could not find enough information that could lead a third party module. The stacktrace changes a little bit between crashes except on LINE facebook::jsi::RuntimeDecorator::call (decorator.h:340) and above. What we know at this moment:
Can you provide some help to find It on how to find which module it's being executed? |
|
The issue also only happens on iOS 16 and 17. Our app works with user health monitoring so we had to shutdown the iOS version to solve that issue due to thousands of this crash. |
@neildhar can probably provide more detail, but most of the fields are documented here: hermes/include/hermes/VM/GCBase.h Line 365 in ff6e5a9 The other fields just show the total number of GCs and how much time in milliseconds they took. |
Thank you this header file contains all that i need about the getInstrumentedStats. Can you provide some help which files contains the information about which react-native native module is calling the jsi environment? |
|
Found something! This code from watermelon database heavily access the jsi runtime. #include "Database.h"
#include "DatabasePlatform.h"
#include "JSIHelpers.h"
namespace watermelondb {
using platform::consoleError;
using platform::consoleLog;
void Database::install(jsi::Runtime *runtime) {
jsi::Runtime &rt = *runtime;
auto globalObject = rt.global();
createMethod(rt, globalObject, "nativeWatermelonCreateAdapter", 2, [runtime](jsi::Runtime &rt, const jsi::Value *args) {
std::string dbPath = args[0].getString(rt).utf8(rt);
bool usesExclusiveLocking = args[1].getBool();
jsi::Object adapter(rt);
std::shared_ptr<Database> database = std::make_shared<Database>(runtime, dbPath, usesExclusiveLocking);
adapter.setProperty(rt, "database", jsi::Object::createFromHostObject(rt, database));
// FIXME: Important hack!
// Without any hacks, JSI Watermelon crashes on Android/Hermes on app reload in development:
// (This doesn't happen on iOS/JSC)
// abort 0x00007d0bd27cff2f
// __fortify_fatal(char const*, ...) 0x00007d0bd27d20c1
// HandleUsingDestroyedMutex(pthread_mutex_t*, char const*) 0x00007d0bd283b020
// pthread_mutex_lock 0x00007d0bd283aef4
// pthreadMutexEnter sqlite3.c:26320
// sqlite3_mutex_enter sqlite3.c:25775
// sqlite3_next_stmt sqlite3.c:84221
// watermelondb::SqliteDb::~SqliteDb() Sqlite.cpp:57
// It appears that the Unix thread on which Database is set up is already destroyed by the
// time destructor is called. AFAIU destructors on objects that are managed by JSI runtime
// *should* be safe in this respect, but maybe they're not/there's a bug...
//
// For future debuggers, the flow goes like this:
// - ReactInstanceManager.runCreateReactContextOnNewThread()
// this sets up new instance
// - ReactInstanceManager.tearDownReactContext()
// - ReactContext.destroy()
// - CatalystInstanceImpl.destroy()
// this notifies listeners that the app is about to be destroyed
// - mHybridData.resetNative()
// - ~CatalystInstanceImpl()
// - ~Instance()
// - NativeToJSBridge.destroy()
// - m_executor = nullptr
// - ~Runtime()
// - ...
// - ~Database()
//
// First attempt to work around this issue was by disabling sqlite3's threadsafety (which caused
// pthread apis to be called, leading to a crash), since we're only using it from one thread
// but predictably that caused new issues.
// When using headless JS, this issue would occur:
// Failed to get a row for query - sqlite error 11 (database disk image is malformed)
// (Not exactly sure why, seems like headless JS reuses the same catalyst instance...)
//
// Current workaround is to tap into CatalystInstanceImpl.destroy() to destroy the database
// before it's destructed via normal C++ rules. There's no clean API for our JSI setup, so
// we route via NativeModuleRegistry onCatalystInstanceDestroy -> DatabaseBridge ->
// WatermelonJSI via reflection (and switch to the currect thread - important!) and then to
// individual Database objects via this listener callback. It's ugly, but should work.
//
// 2023 update: Check if the above is still true, given https://github.com/Nozbe/WatermelonDB/issues/1474
// showed that the true cause of the pthread_mutex_lock crash is something else.
// On the other hand, it's still true that invalidation happens asynchronously and could happen
// after new bridge is already set up, which could cause locking issues (and a case was found on iOS where
// this does happen)
std::weak_ptr<Database> weakDatabase = database;
platform::onDestroy([weakDatabase]() {
if (auto databaseToDestroy = weakDatabase.lock()) {
consoleLog("Destroying database due to RCTBridge invalidation");
databaseToDestroy->destroy();
}
});
createMethod(rt, adapter, "initialize", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
jsi::String dbName = args[0].getString(rt);
int expectedVersion = (int)args[1].getNumber();
int databaseVersion = database->getUserVersion();
jsi::Object response(rt);
if (databaseVersion == expectedVersion) {
database->initialized_ = true;
response.setProperty(rt, "code", "ok");
} else if (databaseVersion == 0) {
response.setProperty(rt, "code", "schema_needed");
} else if (databaseVersion < expectedVersion) {
response.setProperty(rt, "code", "migrations_needed");
response.setProperty(rt, "databaseVersion", databaseVersion);
} else {
consoleLog("Database has newer version (" + std::to_string(databaseVersion) +
") than what the app supports (" + std::to_string(expectedVersion) + "). Will reset database.");
response.setProperty(rt, "code", "schema_needed");
}
return response;
});
createMethod(rt, adapter, "setUpWithSchema", 3, [database](jsi::Runtime &rt, const jsi::Value *args) {
jsi::String dbName = args[0].getString(rt);
jsi::String schema = args[1].getString(rt);
int schemaVersion = (int)args[2].getNumber();
try {
database->unsafeResetDatabase(schema, schemaVersion);
} catch (const std::exception &ex) {
consoleError("Failed to set up the database correctly - " + std::string(ex.what()));
std::abort();
}
database->initialized_ = true;
return jsi::Value::undefined();
});
createMethod(rt, adapter, "setUpWithMigrations", 4, [database](jsi::Runtime &rt, const jsi::Value *args) {
jsi::String dbName = args[0].getString(rt);
jsi::String migrationSchema = args[1].getString(rt);
int fromVersion = (int)args[2].getNumber();
int toVersion = (int)args[3].getNumber();
try {
database->migrate(migrationSchema, fromVersion, toVersion);
} catch (const std::exception &ex) {
consoleError("Failed to migrate the database correctly - " + std::string(ex.what()));
return makeError(rt, ex.what());
}
database->initialized_ = true;
return jsi::Value::undefined();
});
createMethod(rt, adapter, "find", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
jsi::String tableName = args[0].getString(rt);
jsi::String id = args[1].getString(rt);
return database->find(tableName, id);
});
createMethod(rt, adapter, "query", 3, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
jsi::String tableName = args[0].getString(rt);
jsi::String sql = args[1].getString(rt);
jsi::Array arguments = args[2].getObject(rt).getArray(rt);
return database->query(tableName, sql, arguments);
});
createMethod(rt, adapter, "queryAsArray", 3, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
jsi::String tableName = args[0].getString(rt);
jsi::String sql = args[1].getString(rt);
jsi::Array arguments = args[2].getObject(rt).getArray(rt);
return database->queryAsArray(tableName, sql, arguments);
});
createMethod(rt, adapter, "queryIds", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
jsi::String sql = args[0].getString(rt);
jsi::Array arguments = args[1].getObject(rt).getArray(rt);
return database->queryIds(sql, arguments);
});
createMethod(rt, adapter, "unsafeQueryRaw", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
jsi::String sql = args[0].getString(rt);
jsi::Array arguments = args[1].getObject(rt).getArray(rt);
return database->unsafeQueryRaw(sql, arguments);
});
createMethod(rt, adapter, "count", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
jsi::String sql = args[0].getString(rt);
jsi::Array arguments = args[1].getObject(rt).getArray(rt);
return database->count(sql, arguments);
});
createMethod(rt, adapter, "batch", 1, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
jsi::Array operations = args[0].getObject(rt).getArray(rt);
database->batch(operations);
return jsi::Value::undefined();
});
createMethod(rt, adapter, "batchJSON", 1, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
database->batchJSON(args[0].getString(rt));
return jsi::Value::undefined();
});
createMethod(rt, adapter, "getLocal", 1, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
jsi::String key = args[0].getString(rt);
return database->getLocal(key);
});
createMethod(rt, adapter, "unsafeLoadFromSync", 4, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
auto jsonId = (int) args[0].getNumber();
auto schema = args[1].getObject(rt);
auto preamble = args[2].getString(rt).utf8(rt);
auto postamble = args[3].getString(rt).utf8(rt);
return database->unsafeLoadFromSync(jsonId, schema, preamble, postamble);
});
createMethod(rt, adapter, "unsafeExecuteMultiple", 1, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
auto sqlString = args[0].getString(rt).utf8(rt);
database->executeMultiple(sqlString);
return jsi::Value::undefined();
});
createMethod(rt, adapter, "unsafeResetDatabase", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
jsi::String schema = args[0].getString(rt);
int schemaVersion = (int)args[1].getNumber();
try {
database->unsafeResetDatabase(schema, schemaVersion);
return jsi::Value::undefined();
} catch (const std::exception &ex) {
consoleError("Failed to reset database correctly - " + std::string(ex.what()));
// Partially reset database is likely corrupted, so it's probably less bad to crash
std::abort();
}
});
createMethod(rt, adapter, "unsafeClose", 0, [database](jsi::Runtime &rt, const jsi::Value *args) {
assert(database->initialized_);
database->destroy();
database->initialized_ = false;
return jsi::Value::undefined();
});
return adapter;
});
// TODO: Use the onMemoryAlert hook!
}
} // namespace watermelondb |
That would fall outside Hermes unfortunately. It is possible that RN has some logging that logs transitions into native code from JS, but we can't provide much input there. Your best bet may be to attach a debugger and set breakpoints on entry into the Hermes runtime to determine if anything is potentially corrupting runtime state. |
|
Can you provide some interesting points on React-jsi directory to place the debugger on? |
|
This is probably a good place, you should get a hit whenever native calls into JavaScript: Line 2160 in ff6e5a9 |
|
|
|
@lucaswitch According to Apple's documentation it looks like the leak detector works by scanning the application memory for pointers to regions that were allocated with malloc. The behaviour you're seeing is expected, since we recently made a change to how the pointers are encoded in the heap, which would result in the leaks tool not being able to find them. However, this isn't a real leak, and Hermes should still release the memory whenever the ArrayBuffer is garbage collected. |
|
Thank you for the fast response! Ok i will keep looking around on jsi modules. |
|
The stacktrace changed a bit. Removed a lot of the application deps. |
|
Another crash, also i can see that the Module being called is the JSTimers. |
|
Also i figure out that this is the current third party modules accessing the jsi functions.
|
|
@lucaswitch the top of the stack looks a deadlock exception thrown by the sampling profiler mutex. Very interesting. Can we see the message of that exception? |
|
Here is another one maybe related to #1203. Also i was not able to get the exception pointer value...
|
|
Solved! Closing as this issue is not a hermes engine issue. Thank you for all the support on this issue. @tmikov @neildhar |
Bug Description
The iOS react-native application suddenly crashes when the user are consuming it. Not sure which it's the cause. When those crashes occurs, i can deal because happens usually it's a community, or invalid js call, which i can backtrace it using the stacktrace(somehow the problematic module is always on the stacktrace.), but in this case can't see which module caused it leaving me no choice but search help here since the crash happened on hermes engine.
Having a random issue using the 0.73 version of react-native.
Hermes enabled and new arch disabled.
Hermes version: Not sure which version, the 0.73 react-native bundled hermes version.
React Native: 0.73
OS: 17.1
Platform: arm64-v8a, armeabi-v7a
Also here is my react-native-info:
Full stacktrace:
code example:
The Expected Behavior
Not to crash.
The text was updated successfully, but these errors were encountered: