MongoDB server crashes several times a week #144

adrienjoly · 2018-04-02T10:06:31Z

Crashes were much more frequent when mongodb+nodejs were running on a 2gb-ram instance, than now on a 3gb-ram instance. But still, I need to manually run sudo service mongodb restart several times a week, whenever openwhyd users see a "502 Bad Gateway" page while trying to access openwhyd.org. (e.g. https://www.facebook.com/groups/openwhyd/permalink/2056648404616708/)

Hypothesis:

whydjs (our nodejs app) sometimes takes too much memory (e.g. when running data joints in memory) too quickly, so Mongodb just panics
the system is not configurated adequately for mongodb
mongodb could run entirely from our SSD drive --> no need for RAM?
could upgrading to the latest version of mongodb solve this issue?

Next steps:

download and read /var/log/mongodb/mongod.log
apply mongodb's "warning" recommendations (Disable Transparent Huge Pages (THP)) then reboot droplet
find solutions on how to reduce mongodb memory usage (to try if it crashes again)
wait 1 week to check if applying mongodb's recommendations prevented crashes, or try other solutions
if above solutions don't fix the issue, check out mongodb's changelog and consider upgrading
update openwhyd's server maintainance cookbook, to reflect changes

The text was updated successfully, but these errors were encountered:

adrienjoly · 2018-04-02T10:19:34Z

Notes from reading mongodb log:

Some context

2018-02-23T10:56:56.232+0000 I CONTROL  [initandlisten] db version v3.2.19
2018-02-23T10:56:56.232+0000 I CONTROL  [initandlisten] git version: a9f574de6a566a58b24d126b44a56718d181e989
2018-02-23T10:56:56.232+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-02-23T10:56:56.232+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-02-23T10:56:56.232+0000 I CONTROL  [initandlisten] modules: none
2018-02-23T10:56:56.232+0000 I CONTROL  [initandlisten] build environment:
2018-02-23T10:56:56.232+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-02-23T10:56:56.232+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-02-23T10:56:56.232+0000 I CONTROL  [initandlisten]     target_arch: x86_64

and

2018-02-23T10:56:56.289+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-02-23T10:56:56.289+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2018-02-23T10:56:56.518+0000 I STORAGE  [initandlisten] WiredTiger [1519383416:518609][4148:0x7ff494239c80], txn-recover: Main recovery loop: starting at 33/6784
2018-02-23T10:56:56.652+0000 I STORAGE  [initandlisten] WiredTiger [1519383416:652193][4148:0x7ff494239c80], txn-recover: Recovering log 33 through 34
2018-02-23T10:56:56.653+0000 I STORAGE  [initandlisten] WiredTiger [1519383416:653885][4148:0x7ff494239c80], txn-recover: Recovering log 34 through 34
2018-02-23T10:56:56.988+0000 I CONTROL  [initandlisten] 
2018-02-23T10:56:56.988+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-02-23T10:56:56.988+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-02-23T10:56:56.988+0000 I CONTROL  [initandlisten] 
2018-02-23T10:56:56.988+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-02-23T10:56:56.988+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'

Errors

2018-02-23T07:33:02.460+0000 I COMMAND  [conn15] warning: log line attempted (506kB) over max size (10kB), printing beginning and end ... command openwhyd_data.post command: find { find: "post", filter: { uId: { $in: [ "51220e9d7e91c862b2aae21f", "52303f6c7e91c862b2b01744", "507bdefc7e91c862b2a83028", "51e91bea7e91c862b2af3cba", "5152e4d57e91c862b2ac5e3c", "5237187e7e91c862b2b02c2a", "517408fe7e91c862b2ad633b", "51f18c6a7e91c862b2af688a", "5239d8477e91c862b2b034dd", "5239a1667e91c862b2b0336c", "51f2a4807e91c862b2af6c87", "523723367e91c862b2b02cc7", "51a6183f7e91c862b2ae604a", "5077ee747e91c862b2a82e71", "512658df7e91c862b2ab0c28", "51835e047e91c862b2adcee6", "514c12b07e91c862b2abf122", "506eab5b7e91c862b2a82504", "50756a887e91c862b2a82a7a", "514b74017e91c862b2abc602", "52262a7f7e91c862b2affd26", "515306b47e91c862b2ac5f2a", "50bdde567e91c862b2a923dc", "51bb15467e91c862b2aeacfa", "51b2d7c97e91c862b2ae932d", "520785367e91c862b2afac65", "519b96367e91c862b2ae36aa", "5151cb707e91c862b2ac55bc", "514b807b7e91c862b2abcf59", "51e932b57e91c862b2af4c19", "517390b97e91c862b2ad5cdf", "504db7837e91c862b2a80857", "4fb118c368b1a410ecdc0058", "503b49027e91c862b2a8024e", "521e70f17e91c862b2afeb08", "51cc59d87e91c862b2aef465", "51f0d48a7e91c862b2af6567", "5106755d7e91c862b2aa7b28", "52139ec27e91c862b2afcb4c", "4e0c34b7db28e240e8508df3", "515b0a597e91c862b2aca5cf", "521bb8047e91c862b2afe28b", "51094e1c7e91c862b2aa9271", "518cde8b7e91c862b2adfd4c", "521791d57e91c862b2afd7be", "512212dc7e91c862b2aae390", "52182e597e91c862b2afd8fb", "5202d1b57e91c862b2af9e94", "514c34ea7e91c862b2abf6bc", "52123ffe7e91c862b2afc6ad", "51ff77557e91c862b2af8f12", "50bcaf297e91c862b2a91d7c", "52026b567e91c862b2af9c07", "51b7150e7e91c862b2ae9e94", "4dd688ffdb28e240e8508c7e", "514b724a7e91c862b2abc47c", "520a08a37e91c862b2afb32a", "51927cc17e91c862b2ae1680", "4da39da6e98dc192ab1f31d2", "52291ebb7e91c862b2b00485", "522873257e91c862b2b0028d", "521e27387e91c862b2afe9ff", "524301327e91c862b2b0493b", "5226b68c7e91c862b2affe19", "5208dfa47e91c862b2afafd2", "505b15c17e91c862b2a8187d", "511e42e87e91c862b2aad404", "50562cb67e91c862b2a8122c", "5172134a7e91c862b2ad3da5", "51f3e0717e91c862b2af7039", "50be44b67e91c862b2a925c6", "50aa76d87e91c862b2a8b237", "51f3b9727e91c862b2af6f80", "524bd0e07e91c862b2b05c3f", "523237de7e91c862b2b01ff3", "510fd8b87e91c862b2aaad8a", "5158cfcb7e91c862b2ac9358", "51880cf07e91c862b2addef1", "5072efe47e91c862b2a827ce", "50a3afce7e91c862b2a879be", "525589cc7e91c862b2b07316", "50b5e36d7e91c862b2a8f602", "525514307e91c862b2b07145", "505852cc7e91c862b2a814d7", "5252b90e7e91c862b2b06b33", "523861897e91c862b2b03026", "50ebe57f7e91c862b2a9770d", "51c17d747e91c862b2aecfff", "51f599d57e91c862b2af7527", "518beb2c7e91c862b2adf7d8", "516621ad7e91c862b2acdd9e", "5260cfd67e91c862b2b090b3", "5123ff767e91c862b2ab006a", "525171637e91c862b2b068a0", "51581b047e91c862b2ac8bc7", "4d94501d1f78ac091dbc9b4d", "525bcbff7e91c862b2b07fb9", "525c03637e91c862b2b080b1", "51f8c00a7e91c862b2af7e4c", "5252c0127e91c862b2b06b99", "51f7d8d77e91c862b2af7b94", "50ae965f7e91c862b2a8d9be", "51b8437d7e91c862b2aea371", "5267bdb67e91c862b2b0a545", "50a34e867e91c862b2a8724f", "50db4c3a7e91c862b2a96565", "51f2d5777e91c862b2af6d99", "5267dd0d7e91c862b2b0a5d0", "517631df7e91c862b2ad8dc0", "524af4e87e91c862b2b05a7b", "514751317e91c862b2ab624a", "526ec5597e91c862b2b0b7b1", "526069777e91c862b2b09052", "517d730a7e91c862b2adb713", "515d7f047e91c862b2acb6b1", "524054057e91c862b2b04231", "526e6da67e91c862b2b0b520", "525bfe9f7e91c862b2b0809c", "52739bdd7e91c862b2b0c5c4",  .......... c7", "5a5a894f6605de494ce27784", "5a5ae03a6605de494ce277d9", "5a5b3c8e6605de494ce2781b", "5a5b46d26605de494ce27834", "5a5b47886605de494ce27842", "5a5ba63f6605de494ce278ad", "5a5c7e456605de494ce27985", "5a5c7adf6605de494ce27969", "5a5c79926605de494ce27933", "5a5c79286605de494ce2791d", "5a5d5a4a6605de494ce27a40", "5a5de0936605de494ce27afd", "5a5de3666605de494ce27b0b", "5a5dddee6605de494ce27a96", "5a5dd4916605de494ce27a82", "5a5e12b46605de494ce27b50", "5a5e5c216605de494ce27baf", "5a5ee5406605de494ce27c08", "5a5fda8f6605de494ce27c95", "5a6034426605de494ce27cae", "5a605b186605de494ce27cd7", "5a6096ba6605de494ce27d14", "5a60c1896605de494ce27d58", "5a60cdac6605de494ce27d66", "5a60e3a06605de494ce27d85", "5a6207ad6605de494ce27e19", "5a63355a6605de494ce27e97", "5a63490e6605de494ce27eaf", "5a6354946605de494ce27ed3", "5a63a62f6605de494ce27f20", "5a648a006605de494ce27f8d", "5a648f146605de494ce27f9f", "5a65061f6605de494ce2803b", "5a64da6c6605de494ce28009", "5a6589066605de494ce28074", "5a6615526605de494ce28113", "5a660b876605de494ce28107", "5a669ab76605de494ce28164", "5a6681b66605de494ce28150", "5a665ac86605de494ce2813b", "5a664a8b6605de494ce28130", "5a676ba26605de494ce2821f", "5a67b8336605de494ce28262", "5a6778236605de494ce2822b", "5a68e70a6605de494ce28314", "5a68d2406605de494ce282fb", "5a69cbc76605de494ce283ad", "5a6a9ce66605de494ce28439", "5a6a7ce66605de494ce28429", "5a6a29ff6605de494ce283f3", "5a6ae3316605de494ce28452", "5a6d1a526605de494ce285d4", "5a6f4a446605de494ce286d9", "5a6fa4516605de494ce2871a", "5a6f79576605de494ce28700", "5a7115d56605de494ce287da", "5a70bb406605de494ce287a7", "5a727d4e6605de494ce28863", "5a736c176605de494ce288ea", "5a7350546605de494ce288cc", "5a74ba1d6605de494ce28950", "5a74b2766605de494ce28944", "5a74aa1e6605de494ce28937", "5a7495336605de494ce2892c", "5a75a8246605de494ce289a1", "5a75f6276605de494ce28a02", "5a7689e56605de494ce28a49", "5a76d6dc6605de494ce28a55", "5a77388b6605de494ce28b19", "5a77a9eb6605de494ce28b59", "5a77a14f6605de494ce28b4e", "5a7898966605de494ce28bd1", "5a78d3e26605de494ce28c18", "5a7a6d4b6605de494ce28cb8", "5a7c96dc6605de494ce28d82", "5a0f8a706605de494ce248ca", "5a7d5b7e6605de494ce28dbb", "5a7e2b596605de494ce28e1b", "5a7ee1016605de494ce28e6d", "5a8066006605de494ce28f6a", "5a81f8be6605de494ce29012", "5a81dd166605de494ce28ffe", "5a8382b96605de494ce29103", "5a83d4946605de494ce2911c", "5a83f1fc6605de494ce2912e", "5a8539fe6605de494ce29192", "5a8571ca6605de494ce291ed", "5a85665c6605de494ce291e2", "5a85dd4f6605de494ce29237", "5a87e2eb6605de494ce292bd", "5a8784156605de494ce292a6", "5a86e5456605de494ce2927a", "5a86b7d96605de494ce29263", "5a88676a6605de494ce29307", "5a8933d16605de494ce29368", "5a89775e6605de494ce293ba", "5a8ab3612cbe775def2aad74", "5a8aed572cbe775def2aada2", "5a8af9722cbe775def2aadb1", "5a8bf5d32cbe775def2aae14", "5a8b401f2cbe775def2aade5", "5a8c95a82cbe775def2aae8d", "5a8c8fba2cbe775def2aae79", "5a8c7d392cbe775def2aae5e", "5a8f2db4910e03eaa7bbf3cf", "594d0de32049b58e32e961b7", "5a8f9ea3910e03eaa7bbf3e6", "518b5a447e91c862b2adea1a" ] } }, sort: { _id: -1 }, batchSize: 100 } planSummary: IXSCAN { _id: 1 } cursorid:65790118413 keysExamined:162 docsExamined:162 fromMultiPlanner:1 replanned:1 keyUpdates:0 writeConflicts:0 numYields:680 nreturned:100 reslen:39460 locks:{ Global: { acquireCount: { r: 1362 } }, Database: { acquireCount: { r: 681 } }, Collection: { acquireCount: { r: 681 } } } protocol:op_query 7026ms
2018-02-23T07:33:06.943+0000 I COMMAND  [conn11] command openwhyd_data.user command: find { find: "user", filter: { pref.pendEN: { $gt: 0 }, pref.nextEN: { $lte: new Date(1519371186726) } }, projection: {} } planSummary: IXSCAN { pref.nextEN: 1.0 } keysExamined:636 docsExamined:636 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:9 nreturned:0 reslen:124 locks:{ Global: { acquireCount: { r: 20 } }, Database: { acquireCount: { r: 10 } }, Collection: { acquireCount: { r: 10 } } } protocol:op_query 188ms
2018-02-23T07:33:08.497+0000 I COMMAND  [conn14] command openwhyd_data.follow command: count { count: "follow", query: { tId: "53793fa971eaec19b57199e8" } } planSummary: COUNT_SCAN { tId: 1.0 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:62 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 119ms
2018-02-23T07:33:09.286+0000 F -        [conn13] out of memory.

 0x1559962 0x15590b9 0x1d87ac2 0xcbbd81 0xf8e3ae 0xf8e5e1 0xf8e99d 0xf8e637 0xf902c1 0xca5964 0xca7bbf 0xf5aaf5 0xf5c997 0xf5d564 0xf15c78 0xf1697c 0xbc59de 0xc7a4a6 0xc7b79b 0xb8686b 0xdb4aaa 0xdb7fb6 0x9c6e80 0x14ff891 0x7f73731ec6ba 0x7f7372f2241d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"1159962","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"11590B9","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"400000","o":"1987AC2","s":"tc_newarray"},{"b":"400000","o":"8BBD81","s":"_ZNSt6vectorIN5mongo19OrderedIntervalListESaIS1_EEaSERKS3_"},{"b":"400000","o":"B8E3AE","s":"_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE"},{"b":"400000","o":"B8E5E1","s":"_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE"},{"b":"400000","o":"B8E99D","s":"_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE"},{"b":"400000","o":"B8E637","s":"_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE"},{"b":"400000","o":"B902C1","s":"_ZN5mongo12StageBuilder5buildEPNS_16OperationContextEPNS_10CollectionERKNS_13QuerySolutionEPNS_10WorkingSetEPPNS_9PlanStageE"},{"b":"400000","o":"8A5964","s":"_ZN5mongo15CachedPlanStage6replanEPNS_15PlanYieldPolicyEb"},{"b":"400000","o":"8A7BBF","s":"_ZN5mongo15CachedPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE"},{"b":"400000","o":"B5AAF5","s":"_ZN5mongo12PlanExecutor12pickBestPlanENS0_11YieldPolicyE"},{"b":"400000","o":"B5C997","s":"_ZN5mongo12PlanExecutor4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EEPKNS_10CollectionERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS0_11YieldPolicyE"},{"b":"400000","o":"B5D564","s":"_ZN5mongo12PlanExecutor4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EEPKNS_10CollectionENS0_11YieldPolicyE"},{"b":"400000","o":"B15C78","s":"_ZN5mongo11getExecutorEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EENS_12PlanExecutor11YieldPolicyEm"},{"b":"400000","o":"B1697C","s":"_ZN5mongo15getExecutorFindEPNS_16OperationContextEPNS_10CollectionERKNS_15NamespaceStringESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS8_EENS_12PlanExecutor11YieldPolicyE"},{"b":"400000","o":"7C59DE","s":"_ZN5mongo7FindCmd3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE"},{"b":"400000","o":"87A4A6","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"400000","o":"87B79B","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"400000","o":"78686B","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"400000","o":"9B4AAA"},{"b":"400000","o":"9B7FB6","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"400000","o":"5C6E80"},{"b":"400000","o":"10FF891","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"7F73731E5000","o":"76BA"},{"b":"7F7372E1B000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.19", "gitVersion" : "a9f574de6a566a58b24d126b44a56718d181e989", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-112-generic", "version" : "#135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D3471F60D12E548887D24DCA4894F009E439F860" }, { "b" : "7FFE07906000", "elfType" : 3, "buildId" : "FA97F4849697BBE252BA1F7FB2316979E93E61DE" }, { "b" : "7F7374171000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7F7373D2D000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7F7373B25000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7F7373921000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7F7373618000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB85DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7F7373402000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F73731E5000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7F7372E1B000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7F73743DA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1559962]
 mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x79) [0x15590b9]
 mongod(tc_newarray+0x252) [0x1d87ac2]
 mongod(_ZNSt6vectorIN5mongo19OrderedIntervalListESaIS1_EEaSERKS3_+0x331) [0xcbbd81]
 mongod(_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE+0x39E) [0xf8e3ae]
 mongod(_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE+0x5D1) [0xf8e5e1]
 mongod(_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE+0x98D) [0xf8e99d]
 mongod(_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE+0x627) [0xf8e637]
 mongod(_ZN5mongo12StageBuilder5buildEPNS_16OperationContextEPNS_10CollectionERKNS_13QuerySolutionEPNS_10WorkingSetEPPNS_9PlanStageE+0x31) [0xf902c1]
 mongod(_ZN5mongo15CachedPlanStage6replanEPNS_15PlanYieldPolicyEb+0x934) [0xca5964]
 mongod(_ZN5mongo15CachedPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE+0x5BF) [0xca7bbf]
 mongod(_ZN5mongo12PlanExecutor12pickBestPlanENS0_11YieldPolicyE+0xC5) [0xf5aaf5]
 mongod(_ZN5mongo12PlanExecutor4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EEPKNS_10CollectionERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS0_11YieldPolicyE+0x287) [0xf5c997]
 mongod(_ZN5mongo12PlanExecutor4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EEPKNS_10CollectionENS0_11YieldPolicyE+0xC4) [0xf5d564]
 mongod(_ZN5mongo11getExecutorEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EENS_12PlanExecutor11YieldPolicyEm+0x108) [0xf15c78]
 mongod(_ZN5mongo15getExecutorFindEPNS_16OperationContextEPNS_10CollectionERKNS_15NamespaceStringESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS8_EENS_12PlanExecutor11YieldPolicyE+0x7C) [0xf1697c]
 mongod(_ZN5mongo7FindCmd3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE+0x8EE) [0xbc59de]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x676) [0xc7a4a6]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x85B) [0xc7b79b]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x25B) [0xb8686b]
 mongod(+0x9B4AAA) [0xdb4aaa]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x7D6) [0xdb7fb6]
 mongod(+0x5C6E80) [0x9c6e80]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x311) [0x14ff891]
 libpthread.so.0(+0x76BA) [0x7f73731ec6ba]
 libc.so.6(clone+0x6D) [0x7f7372f2241d]
-----  END BACKTRACE  -----
2018-02-23T10:56:53.557+0000 I CONTROL  [main] ***** SERVER RESTARTED *****

and

2018-03-31T04:14:58.312+0000 I COMMAND  [conn45] warning: log line attempted (507kB) over max size (10kB), printing beginning and end ... command openwhyd_data.post command: find { find: "post", filter: { uId: { $in: [ "51220e9d7e91c862b2aae21f", "52303f6c7e91c862b2b01744", "507bdefc7e91c862b2a83028", "51e91bea7e91c862b2af3cba", "5152e4d57e91c862b2ac5e3c", "5237187e7e91c862b2b02c2a", "517408fe7e91c862b2ad633b", "51f18c6a7e91c862b2af688a", "5239d8477e91c862b2b034dd", "5239a1667e91c862b2b0336c", "51f2a4807e91c862b2af6c87", "523723367e91c862b2b02cc7", "51a6183f7e91c862b2ae604a", "5077ee747e91c862b2a82e71", "512658df7e91c862b2ab0c28", "51835e047e91c862b2adcee6", "514c12b07e91c862b2abf122", "506eab5b7e91c862b2a82504", "50756a887e91c862b2a82a7a", "514b74017e91c862b2abc602", "52262a7f7e91c862b2affd26", "515306b47e91c862b2ac5f2a", "50bdde567e91c862b2a923dc", "51bb15467e91c862b2aeacfa", "51b2d7c97e91c862b2ae932d", "520785367e91c862b2afac65", "519b96367e91c862b2ae36aa", "5151cb707e91c862b2ac55bc", "514b807b7e91c862b2abcf59", "51e932b57e91c862b2af4c19", "517390b97e91c862b2ad5cdf", "504db7837e91c862b2a80857", "4fb118c368b1a410ecdc0058", "503b49027e91c862b2a8024e", "521e70f17e91c862b2afeb08", "51cc59d87e91c862b2aef465", "51f0d48a7e91c862b2af6567", "5106755d7e91c862b2aa7b28", "52139ec27e91c862b2afcb4c", "4e0c34b7db28e240e8508df3", "515b0a597e91c862b2aca5cf", "521bb8047e91c862b2afe28b", "51094e1c7e91c862b2aa9271", "518cde8b7e91c862b2adfd4c", "521791d57e91c862b2afd7be", "512212dc7e91c862b2aae390", "52182e597e91c862b2afd8fb", "5202d1b57e91c862b2af9e94", "514c34ea7e91c862b2abf6bc", "52123ffe7e91c862b2afc6ad", "51ff77557e91c862b2af8f12", "50bcaf297e91c862b2a91d7c", "52026b567e91c862b2af9c07", "51b7150e7e91c862b2ae9e94", "4dd688ffdb28e240e8508c7e", "514b724a7e91c862b2abc47c", "520a08a37e91c862b2afb32a", "51927cc17e91c862b2ae1680", "4da39da6e98dc192ab1f31d2", "52291ebb7e91c862b2b00485", "522873257e91c862b2b0028d", "521e27387e91c862b2afe9ff", "524301327e91c862b2b0493b", "5226b68c7e91c862b2affe19", "5208dfa47e91c862b2afafd2", "505b15c17e91c862b2a8187d", "511e42e87e91c862b2aad404", "50562cb67e91c862b2a8122c", "5172134a7e91c862b2ad3da5", "51f3e0717e91c862b2af7039", "50be44b67e91c862b2a925c6", "50aa76d87e91c862b2a8b237", "51f3b9727e91c862b2af6f80", "524bd0e07e91c862b2b05c3f", "523237de7e91c862b2b01ff3", "510fd8b87e91c862b2aaad8a", "5158cfcb7e91c862b2ac9358", "51880cf07e91c862b2addef1", "5072efe47e91c862b2a827ce", "50a3afce7e91c862b2a879be", "525589cc7e91c862b2b07316", "50b5e36d7e91c862b2a8f602", "525514307e91c862b2b07145", "505852cc7e91c862b2a814d7", "5252b90e7e91c862b2b06b33", "523861897e91c862b2b03026", "50ebe57f7e91c862b2a9770d", "51c17d747e91c862b2aecfff", "51f599d57e91c862b2af7527", "518beb2c7e91c862b2adf7d8", "516621ad7e91c862b2acdd9e", "5260cfd67e91c862b2b090b3", "5123ff767e91c862b2ab006a", "525171637e91c862b2b068a0", "51581b047e91c862b2ac8bc7", "4d94501d1f78ac091dbc9b4d", "525bcbff7e91c862b2b07fb9", "525c03637e91c862b2b080b1", "51f8c00a7e91c862b2af7e4c", "5252c0127e91c862b2b06b99", "51f7d8d77e91c862b2af7b94", "50ae965f7e91c862b2a8d9be", "51b8437d7e91c862b2aea371", "5267bdb67e91c862b2b0a545", "50a34e867e91c862b2a8724f", "50db4c3a7e91c862b2a96565", "51f2d5777e91c862b2af6d99", "5267dd0d7e91c862b2b0a5d0", "517631df7e91c862b2ad8dc0", "524af4e87e91c862b2b05a7b", "514751317e91c862b2ab624a", "526ec5597e91c862b2b0b7b1", "526069777e91c862b2b09052", "517d730a7e91c862b2adb713", "515d7f047e91c862b2acb6b1", "524054057e91c862b2b04231", "526e6da67e91c862b2b0b520", "525bfe9f7e91c862b2b0809c", "52739bdd7e91c862b2b0c5c4",  .......... ce28074", "5a6615526605de494ce28113", "5a660b876605de494ce28107", "5a669ab76605de494ce28164", "5a6681b66605de494ce28150", "5a665ac86605de494ce2813b", "5a664a8b6605de494ce28130", "5a676ba26605de494ce2821f", "5a67b8336605de494ce28262", "5a6778236605de494ce2822b", "5a68e70a6605de494ce28314", "5a68d2406605de494ce282fb", "5a69cbc76605de494ce283ad", "5a6a9ce66605de494ce28439", "5a6a7ce66605de494ce28429", "5a6a29ff6605de494ce283f3", "5a6ae3316605de494ce28452", "5a6d1a526605de494ce285d4", "5a6f4a446605de494ce286d9", "5a6fa4516605de494ce2871a", "5a6f79576605de494ce28700", "5a7115d56605de494ce287da", "5a70bb406605de494ce287a7", "5a727d4e6605de494ce28863", "5a736c176605de494ce288ea", "5a7350546605de494ce288cc", "5a74ba1d6605de494ce28950", "5a74b2766605de494ce28944", "5a74aa1e6605de494ce28937", "5a7495336605de494ce2892c", "5a75a8246605de494ce289a1", "5a75f6276605de494ce28a02", "5a7689e56605de494ce28a49", "5a76d6dc6605de494ce28a55", "5a77388b6605de494ce28b19", "5a77a9eb6605de494ce28b59", "5a77a14f6605de494ce28b4e", "5a7898966605de494ce28bd1", "5a78d3e26605de494ce28c18", "5a7a6d4b6605de494ce28cb8", "5a7c96dc6605de494ce28d82", "5a0f8a706605de494ce248ca", "5a7d5b7e6605de494ce28dbb", "5a7e2b596605de494ce28e1b", "5a7ee1016605de494ce28e6d", "5a8066006605de494ce28f6a", "5a81f8be6605de494ce29012", "5a81dd166605de494ce28ffe", "5a8382b96605de494ce29103", "5a83d4946605de494ce2911c", "5a83f1fc6605de494ce2912e", "5a8539fe6605de494ce29192", "5a8571ca6605de494ce291ed", "5a85665c6605de494ce291e2", "5a85dd4f6605de494ce29237", "5a87e2eb6605de494ce292bd", "5a8784156605de494ce292a6", "5a86e5456605de494ce2927a", "5a86b7d96605de494ce29263", "5a88676a6605de494ce29307", "5a8933d16605de494ce29368", "5a89775e6605de494ce293ba", "5a8ab3612cbe775def2aad74", "5a8aed572cbe775def2aada2", "5a8af9722cbe775def2aadb1", "5a8bf5d32cbe775def2aae14", "5a8b401f2cbe775def2aade5", "5a8c95a82cbe775def2aae8d", "5a8c8fba2cbe775def2aae79", "5a8c7d392cbe775def2aae5e", "5a8f2db4910e03eaa7bbf3cf", "594d0de32049b58e32e961b7", "5a8f9ea3910e03eaa7bbf3e6", "5a93751695bbab0677572d15", "5a9353d595bbab0677572d05", "5a943bfd95bbab0677572dd6", "5a96f9b91a7e605dccb6a8df", "5a98325a1a7e605dccb6a981", "5a9879c21a7e605dccb6a9a8", "5a996eb91a7e605dccb6a9f4", "5a9a62e61a7e605dccb6aa52", "5a99b68a1a7e605dccb6aa22", "5a9ac79b1a7e605dccb6aa8e", "5a9c0b3c1a7e605dccb6ab63", "5a9d05121a7e605dccb6abd3", "5a9ee1771a7e605dccb6ad01", "5a9faa281a7e605dccb6ad72", "5a9edae21a7e605dccb6acf6", "5aa45daceadbabb9280d0e99", "5aa6ec5deadbabb9280d0fb8", "5aa7d85feadbabb9280d102d", "5aa7dacdeadbabb9280d103f", "5aa9e5bceadbabb9280d110c", "5aaa3614eadbabb9280d1149", "5961278a2049b58e32e9726c", "5aabe0d5eadbabb9280d11bc", "5aabb6efeadbabb9280d11a6", "5aabb4aeeadbabb9280d119b", "5aaef726eadbabb9280d132b", "5ab1963fe9ad8b9b12282e17", "5ab164e7e9ad8b9b12282e00", "5ab37c20e9ad8b9b12282ebf", "5aba0796e9ad8b9b1228313d", "5abb2203e9ad8b9b122831d1", "5abc3b07e9ad8b9b1228323b", "5abede7de9ad8b9b122832e6", "518b5a447e91c862b2adea1a" ] }, _id: { $lt: ObjectId('5aba90615e4d5b5c85063316') } }, sort: { _id: -1 }, batchSize: 100 } planSummary: IXSCAN { _id: 1 } cursorid:64926264961 keysExamined:224 docsExamined:224 fromMultiPlanner:1 replanned:1 keyUpdates:0 writeConflicts:0 numYields:650 nreturned:100 reslen:37693 locks:{ Global: { acquireCount: { r: 1302 } }, Database: { acquireCount: { r: 651 } }, Collection: { acquireCount: { r: 651 } } } protocol:op_query 590ms
2018-03-31T04:15:07.581+0000 I COMMAND  [conn42] command openwhyd_data.post command: aggregate { aggregate: "post", pipeline: [ { $match: { uId: "518b5a447e91c862b2adea1a" } }, { $group: { _id: "$pl.id", c: { $sum: 1 } } } ], cursor: { batchSize: 1000 } } keyUpdates:0 writeConflicts:0 numYields:1135 reslen:435 locks:{ Global: { acquireCount: { r: 2298 } }, Database: { acquireCount: { r: 1149 } }, Collection: { acquireCount: { r: 1149 } } } protocol:op_query 1426ms
2018-03-31T04:15:08.129+0000 I COMMAND  [conn41] command openwhyd_data.follow command: find { find: "follow", filter: { uId: null, tId: "518b5a447e91c862b2adea1a" }, limit: 1, singleBatch: true, batchSize: 1 } planSummary: IXSCAN { tId: 1.0 } keysExamined:16967 docsExamined:16967 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:134 nreturned:0 reslen:126 locks:{ Global: { acquireCount: { r: 270 } }, Database: { acquireCount: { r: 135 } }, Collection: { acquireCount: { r: 135 } } } protocol:op_query 398ms
2018-03-31T04:15:09.715+0000 I COMMAND  [conn43] command openwhyd_data.post command: count { count: "post", query: { uId: "518b5a447e91c862b2adea1a", rTo: null } } planSummary: IXSCAN { uId: 1.0, pl.id: 1.0 } keyUpdates:0 writeConflicts:0 numYields:1139 reslen:62 locks:{ Global: { acquireCount: { r: 2280 } }, Database: { acquireCount: { r: 1140 } }, Collection: { acquireCount: { r: 1140 } } } protocol:op_query 1509ms
2018-03-31T04:15:10.050+0000 I COMMAND  [conn42] command openwhyd_data.post command: find { find: "post", filter: { uId: "518b5a447e91c862b2adea1a", pl.id: 0 }, sort: { order: 1, _id: -1 }, limit: 21 } planSummary: IXSCAN { uId: 1.0, pl.id: 1.0 } keysExamined:13898 docsExamined:13898 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:109 nreturned:21 reslen:7550 locks:{ Global: { acquireCount: { r: 220 } }, Database: { acquireCount: { r: 110 } }, Collection: { acquireCount: { r: 110 } } } protocol:op_query 241ms
2018-03-31T04:15:13.491+0000 F -        [conn42] out of memory.

 0x1559962 0x15590b9 0x14d50a1 0x9cedfb 0x12f3098 0xc79e77 0xc7b79b 0xb8686b 0xdb4aaa 0xdb7fb6 0x9c6e80 0x14ff891 0x7fda0bb666ba 0x7fda0b89c41d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"1159962","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"11590B9","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"400000","o":"10D50A1","s":"_ZN5mongo12mongoReallocEPvm"},{"b":"400000","o":"5CEDFB","s":"_ZN5mongo11_BufBuilderINS_16TrivialAllocatorEE15grow_reallocateEi"},{"b":"400000","o":"EF3098","s":"_ZN5mongo3rpc18LegacyReplyBuilder22getInPlaceReplyBuilderEm"},{"b":"400000","o":"879E77","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"400000","o":"87B79B","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"400000","o":"78686B","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"400000","o":"9B4AAA"},{"b":"400000","o":"9B7FB6","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"400000","o":"5C6E80"},{"b":"400000","o":"10FF891","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"7FDA0BB5F000","o":"76BA"},{"b":"7FDA0B795000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.19", "gitVersion" : "a9f574de6a566a58b24d126b44a56718d181e989", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-116-generic", "version" : "#140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D3471F60D12E548887D24DCA4894F009E439F860" }, { "b" : "7FFDD554F000", "elfType" : 3, "buildId" : "2743AEADCDA27954EA08FA4C94C6DB97E2A37F92" }, { "b" : "7FDA0CAEB000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7FDA0C6A7000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7FDA0C49F000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7FDA0C29B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7FDA0BF92000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB85DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7FDA0BD7C000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FDA0BB5F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7FDA0B795000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7FDA0CD54000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1559962]
 mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x79) [0x15590b9]
 mongod(_ZN5mongo12mongoReallocEPvm+0x21) [0x14d50a1]
 mongod(_ZN5mongo11_BufBuilderINS_16TrivialAllocatorEE15grow_reallocateEi+0x3B) [0x9cedfb]
 mongod(_ZN5mongo3rpc18LegacyReplyBuilder22getInPlaceReplyBuilderEm+0x38) [0x12f3098]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x47) [0xc79e77]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x85B) [0xc7b79b]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x25B) [0xb8686b]
 mongod(+0x9B4AAA) [0xdb4aaa]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x7D6) [0xdb7fb6]
 mongod(+0x5C6E80) [0x9c6e80]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x311) [0x14ff891]
 libpthread.so.0(+0x76BA) [0x7fda0bb666ba]
 libc.so.6(clone+0x6D) [0x7fda0b89c41d]
-----  END BACKTRACE  -----
2018-03-31T12:37:51.701+0000 I CONTROL  [main] ***** SERVER RESTARTED *****

adrienjoly · 2018-04-02T10:44:36Z

Applied mongodb's "warning" recommendations (Disable Transparent Huge Pages (THP)).

After rebooting the droplet, the warning disappears from mongodb's logs! 👌

adrienjoly · 2018-04-02T10:54:02Z

About having mongodb to use only SDD instead of RAM:

" I think SSDs are great for database performance all around but again my experience has been that you're better off making certain your working set remains in memory. "

(source: https://stackoverflow.com/a/23481724/592254)

adrienjoly · 2018-04-02T10:56:14Z

Solutions to reduce mongob RAM usage (if still necessary):

Reduce size of WiredTiger internal cache

In MongoDB 3.2, the WiredTiger internal cache, by default, will use the larger of either: 60% of RAM minus 1 GB, or 1 GB.

=> 60% * 3 - 1 = 1.8 -1 = 0.8 GB on our current (3gb-ram) instance

It's possible to manually set this using the storage.wiredTiger.engineConfig.cacheSizeGB config parameter, as documented there: Configuration File Options — MongoDB Manual 3.2.

Limit mongod process usage using cgroups

source: https://stackoverflow.com/a/34805475/592254

adrienjoly · 2018-04-02T11:12:19Z

Charts of last 7 days:

=> Conclusions after disabling transparent huge page:

Average RAM usage dropped from 78% to 54%. (for now)
CPU usage remains stable at ~3%

Let's see how in goes, in a few days.

adrienjoly · 2018-04-14T23:19:31Z

The server has not crashed for 2 weeks! 🎉

adrienjoly · 2020-03-01T17:21:48Z

It crashed again on February 22nd, 2020, while I was in vacation in Scotland => I activated a process monitor on Datadog

coty-crg · 2020-06-02T23:34:26Z

Just letting you know, you're not alone. I have my scripts just auto-restart mongo at this point; it randomly crashes after running for long periods of time! I can only assume it's due to the small amount of memory available to it, but that's not an acceptable reason.

adrienjoly · 2020-06-03T16:50:13Z

Thanks for sharing your experience, Coty, it really makes me feel better to see that we're not alone to experience this!

The fact that you also run it on a machine with a tight amount of memory kinds of confirm that lack of memory could be the root cause of this problem.

Thank you and have a great week!

adrienjoly · 2020-06-13T13:47:11Z

Makes sense:

Related: #202

Contributes to #318, #202 and #144. cf #318 (comment) * remove /styles pages * remove /genre routes

## [1.37.7](v1.37.6...v1.37.7) (2020-11-10) ### Bug Fixes * Remove slow routes ([#386](#386)) ([0ac5e41](0ac5e41)), closes [#318](#318) [#202](#202) [#144](#144) [/github.com//issues/318#issuecomment-724748867](https://github.com//github.com/openwhyd/openwhyd/issues/318/issues/issuecomment-724748867)

arpan1011 · 2021-11-30T10:53:30Z

Just letting you know, you're not alone. I have my scripts just auto-restart mongo at this point; it randomly crashes after running for long periods of time! I can only assume it's due to the small amount of memory available to it, but that's not an acceptable reason.

Are these scripts written inside node ( I mean can I restart mongo service when I know it has crashed ) ?

adrienjoly · 2021-11-30T11:24:44Z

I know it does not answer your question, but it may help you to know that my MongoDB instance has been running without interruptions since I made sure to close cursors after opening them, see https://github.com/openwhyd/openwhyd/pull/405/files#diff-5ad5ab59edde086d4dc1e2ad75d0637631903bb6ae760e7581ee7ea1dc0ed961R147.

adrienjoly created this issue from a note in Development (⚙ In progress) Apr 2, 2018

adrienjoly self-assigned this Apr 2, 2018

adrienjoly added the bug label Apr 2, 2018

adrienjoly moved this from ⚙ In progress to ✔️ Done / pending QA in Development Apr 2, 2018

adrienjoly closed this as completed Apr 14, 2018

adrienjoly moved this from ✔️ Done / pending QA to 🌲 In production in Development Apr 14, 2018

adrienjoly mentioned this issue Apr 15, 2018

Where to store usage logs and analytics? #138

Closed

3 tasks

adrienjoly mentioned this issue Sep 15, 2018

MongoDB crashes ~once/week #162

Closed

2 tasks

This was referenced Nov 10, 2020

Prevent MongoDB crashes, by reducing the number of opened cursors #318

Closed

fix: Remove slow routes #386

Merged

adrienjoly added a commit that referenced this issue Nov 10, 2020

fix: Remove slow routes (#386)

0ac5e41

Contributes to #318, #202 and #144. cf #318 (comment) * remove /styles pages * remove /genre routes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MongoDB server crashes several times a week #144

MongoDB server crashes several times a week #144

adrienjoly commented Apr 2, 2018 •

edited

Loading

adrienjoly commented Apr 2, 2018

adrienjoly commented Apr 2, 2018

adrienjoly commented Apr 2, 2018

adrienjoly commented Apr 2, 2018 •

edited

Loading

adrienjoly commented Apr 2, 2018

adrienjoly commented Apr 14, 2018

adrienjoly commented Mar 1, 2020

coty-crg commented Jun 2, 2020

adrienjoly commented Jun 3, 2020

adrienjoly commented Jun 13, 2020

arpan1011 commented Nov 30, 2021

adrienjoly commented Nov 30, 2021

MongoDB server crashes several times a week #144

MongoDB server crashes several times a week #144

Comments

adrienjoly commented Apr 2, 2018 • edited Loading

adrienjoly commented Apr 2, 2018

Some context

Errors

adrienjoly commented Apr 2, 2018

adrienjoly commented Apr 2, 2018

adrienjoly commented Apr 2, 2018 • edited Loading

Reduce size of WiredTiger internal cache

Limit mongod process usage using cgroups

adrienjoly commented Apr 2, 2018

adrienjoly commented Apr 14, 2018

adrienjoly commented Mar 1, 2020

coty-crg commented Jun 2, 2020

adrienjoly commented Jun 3, 2020

adrienjoly commented Jun 13, 2020

arpan1011 commented Nov 30, 2021

adrienjoly commented Nov 30, 2021

adrienjoly commented Apr 2, 2018 •

edited

Loading

adrienjoly commented Apr 2, 2018 •

edited

Loading