ArangoDB reload taking lot of time when pod restarts

[yogeshbidari]

My Environment

• Storage Engine: RocksDB
• Deployment Mode: single server
• Deployment Strategy: kuberentes
• Infrastructure: GKE
• Operating System: Debian 9
• Total RAM in your machine: 180GB
• Disks in use: SSD
• Used Package: Docker

Component, Query & Data

Affected feature:

AQL query (if applicable):

AQL explain and/or profile (if applicable):

Dataset:

Size of your Dataset on disk:
80GB

Replication Factor & Number of Shards (Cluster only):

Steps to reproduce

1. When POD restarts because of any issue it takes lot of time to reload the databases, we have around 16k DBs total of 80GB in size.

Problem:
After Pod restart, it takes around 16hrs to reload the databases, till this time will not be able to access arango service in both CLI and UI.

We see some slow query logs in POD logs
2021-06-17T23:44:05Z [1] WARNING {queries} slow query: '\n FOR item IN @value0\n UPSERT {mount: item[0]}\n INSERT item[1]\n REPLACE item[1]\n IN @@value1\n ', bind vars: {"@value1":"_apps","value0":[["/_admin/aardvark",{"id":"/_admin/aardvark","mount":"/_admin/aardvark","options":{"configuration":{},"dependencies":{},"development":false}}],["/_api/foxx",{"id":"/_api/foxx","mount":"/_api/foxx","options":{"configuration":{},"dependencies":{},"development":false}}],["/_api/gharial",{"id":"/_api/gharial","mount":"/_api/gharial","options":{"configuration":{},"dependencies":{},"development":false}}]]}, took: 18.012608 s

We have enabled trace logging also, but we dont see any extra logs except slow query and below one
2021-06-18T15:58:58Z [1] TRACE optimization ends with 1 plans 2021-06-18T15:58:58Z [1] TRACE optimization ends with 1 plans 2021-06-18T15:58:58Z [1] TRACE optimization ends with 1 plans 2021-06-18T15:58:58Z [1] TRACE optimization ends with 1 plans

Does reloading of 16k or 80GB databases after pod restart takes so much of time?

Expected result:
DB reload has to take less time.

[Simran-B]

@yogeshbidari Can you share a bit more information? In particular the K8s configuration?

Do I understand correctly that the two log messages that you posted are the only output from arangod? How exactly did you active trace logging? With --log.level trace? I just tried that locally and it seems like that this does not active trace logging for all log topics, but just for "general" log messages. You could try --log.level startup=trace (and/or perhaps engines, rocksdb) and see if that gives any hints to what it is busy with.

[Simran-B]

I checked with engineering and they clarified that --log.level <level> controls the general log topic only. You need to set individual log levels per topics if you want more than just the general trace logs. There is a list of log topics here: https://www.arangodb.com/docs/stable/http/administration-and-monitoring.html#modify-and-return-the-current-server-log-level

[yogeshbidari]

@Simran-B Yes, I am using --log.level trace as an argument. I will set the individual log levels and try. Let you know on the status.

[Simran-B]

@yogeshbidari Did you find anything in the trace logs yet?

[yogeshbidari]

@Simran-B I have changed the log.level to startup=trace , I didn't find any specific errors except lot of slow queries.

Still it is taking lot of time to reload databases.
Sharing the logs below as I am not able to upload log file.

choosing Rocksdb storage engine
{startup} Action::loadOptions
{startup} FileDescriptors::loadOptions
{startup} Greetings::loadOptions
{startup} LoggerBuffer::loadOptions
{startup} Nonce::loadOptions
{startup} PageSize::loadOptions
{startup} Privilege::loadOptions
{startup} Random::loadOptions
{startup} Authentication::loadOptions
{startup} Scheduler::loadOptions
{startup} CacheManager::loadOptions
{startup} Config::loadOptions
{startup} Daemon::loadOptions
{startup} Environment::loadOptions
{startup} Frontend::loadOptions
{startup} Language::loadOptions
{startup} Ssl::loadOptions
{startup} SslServer::loadOptions
{startup} Supervisor::loadOptions
{startup} Temp::loadOptions
{startup} DatabasePath::loadOptions
{startup} EngineSelector::loadOptions
{startup} InitDatabase::loadOptions
{startup} Lockfile::loadOptions
{startup} RocksDBOption::loadOptions
{startup} MMFilesPersistentIndex::loadOptions
{startup} ReplicationTimeout::loadOptions
{startup} TransactionManager::loadOptions
{startup} MMFilesEngine::loadOptions
{startup} RocksDBEngine::loadOptions
{startup} StorageEngine::loadOptions
{startup} Database::loadOptions
{startup} ServerId::loadOptions
{startup} Replication::loadOptions
{startup} CheckVersion::loadOptions
{startup} MMFilesLogfileManager::loadOptions
{startup} MMFilesWalRecovery::loadOptions
{startup} RocksDBRecoveryFinalizer::loadOptions
{startup} TraverserEngineRegistry::loadOptions
{startup} V8Platform::loadOptions
{startup} V8Dealer::loadOptions
{startup} Cluster::loadOptions
{startup} FoxxQueues::loadOptions
{startup} QueryRegistry::loadOptions
{startup} Aql::loadOptions
{startup} AQLFunctions::loadOptions
{startup} OptimizerRules::loadOptions
{startup} Statistics::loadOptions
{startup} Upgrade::loadOptions
{startup} Server::loadOptions
{startup} Endpoint::loadOptions
{startup} Agency::loadOptions
{startup} GeneralServer::loadOptions
{startup} Bootstrap::loadOptions
{startup} Console::loadOptions
{startup} Pregel::loadOptions
{startup} Script::loadOptions
{startup} UnitTests::loadOptions
{startup} Shutdown::loadOptions
{startup} ViewTypes::loadOptions
{startup} ApplicationServer::validateOptions
{startup} Jemalloc::validateOptions
{startup} MMFilesCompaction::validateOptions
{startup} MaxMapCount::validateOptions
{startup} ShellColors::validateOptions
{startup} Version::validateOptions
{startup} WorkMonitor::validateOptions
{startup} Logger::validateOptions
{startup} Action::validateOptions
{startup} FileDescriptors::validateOptions
{startup} Greetings::validateOptions
{startup} LoggerBuffer::validateOptions
{startup} Nonce::validateOptions
{startup} PageSize::validateOptions
{startup} Privilege::validateOptions
{startup} Random::validateOptions
{startup} Authentication::validateOptions
{startup} Scheduler::validateOptions
{startup} CacheManager::validateOptions
{startup} Config::validateOptions
{startup} Daemon::validateOptions
{startup} Environment::validateOptions
{startup} Frontend::validateOptions
{startup} Language::validateOptions
{startup} Ssl::validateOptions
{startup} SslServer::validateOptions
{startup} Supervisor::validateOptions
{startup} Temp::validateOptions
{startup} DatabasePath::validateOptions
{startup} EngineSelector::validateOptions
{startup} InitDatabase::validateOptions
{startup} Lockfile::validateOptions
{startup} RocksDBOption::validateOptions
{startup} MMFilesPersistentIndex::validateOptions
{startup} ReplicationTimeout::validateOptions
{startup} TransactionManager::validateOptions
{startup} MMFilesEngine::validateOptions
{startup} RocksDBEngine::validateOptions
{startup} StorageEngine::validateOptions
{startup} Database::validateOptions
{startup} ServerId::validateOptions
{startup} Replication::validateOptions
{startup} CheckVersion::validateOptions
{startup} MMFilesLogfileManager::validateOptions
{startup} MMFilesWalRecovery::validateOptions
{startup} RocksDBRecoveryFinalizer::validateOptions
{startup} TraverserEngineRegistry::validateOptions
{startup} V8Platform::validateOptions
{startup} V8Dealer::validateOptions
{startup} Cluster::validateOptions
{startup} FoxxQueues::validateOptions
{startup} QueryRegistry::validateOptions
{startup} Aql::validateOptions
{startup} AQLFunctions::validateOptions
{startup} OptimizerRules::validateOptions
{startup} Statistics::validateOptions
{startup} Upgrade::validateOptions
{startup} Server::validateOptions
{startup} Endpoint::validateOptions
{startup} Agency::validateOptions
{startup} GeneralServer::validateOptions
{startup} Bootstrap::validateOptions
{startup} Console::validateOptions
{startup} Pregel::validateOptions
{startup} Script::validateOptions
{startup} UnitTests::validateOptions
{startup} ViewTypes::validateOptions
{startup} ApplicationServer::validateDependencies
{startup} ordered features:
{startup} feature #1: Jemalloc
{startup} feature #2: MMFilesCompaction
{startup} feature #3: MaxMapCount
{startup} feature #4: ShellColors
{startup} feature #5: Version - depends on: ShellColors
{startup} feature #6: WorkMonitor - depends on: ShellColors
{startup} feature #7: Logger - depends on: Version, WorkMonitor, ShellColors
{startup} feature #8: Action - depends on: Logger
{startup} feature #9: FileDescriptors - depends on: Logger
{startup} feature #10: Greetings - depends on: Logger
{startup} feature #11: LoggerBuffer - depends on: Logger
{startup} feature #12: Nonce - depends on: Logger
{startup} feature #13: PageSize - depends on: Logger
{startup} feature #14: Privilege - depends on: Logger
{startup} feature #15: Random - depends on: Logger
{startup} feature #16: Authentication - depends on: Random
{startup} feature #17: Scheduler - depends on: WorkMonitor, Random, Logger, FileDescriptors
{startup} feature #18: CacheManager - depends on: Scheduler
{startup} feature #19: Config - depends on: ShellColors, Logger
{startup} feature #20: Daemon - depends on: WorkMonitor, Logger
{startup} feature #21: Environment - depends on: MaxMapCount, Logger, Greetings
{startup} feature #22: Frontend - depends on: Logger
{startup} feature #23: Language - depends on: Logger
{startup} feature #24: Ssl - depends on: Greetings, Logger
{startup} feature #25: SslServer - depends on: Logger, Ssl
{startup} feature #26: Supervisor - depends on: WorkMonitor, Logger, Daemon
{startup} feature #27: Temp - depends on: Random, Logger
{startup} feature #28: DatabasePath - depends on: WorkMonitor, Temp, Random, FileDescriptors, Language, Logger, PageSize
{startup} feature #29: EngineSelector - depends on: Logger, Greetings, DatabasePath
{startup} feature #30: InitDatabase - depends on: DatabasePath, Logger
{startup} feature #31: Lockfile - depends on: DatabasePath
{startup} feature #32: RocksDBOption - depends on: DatabasePath, Daemon
{startup} feature #33: MMFilesPersistentIndex - depends on: RocksDBOption
{startup} feature #34: ReplicationTimeout - depends on: EngineSelector
{startup} feature #35: TransactionManager - depends on: WorkMonitor, EngineSelector
{startup} feature #36: MMFilesEngine - depends on: MMFilesPersistentIndex, TransactionManager, CacheManager, DatabasePath, FileDescriptors, Temp
{startup} feature #37: RocksDBEngine - depends on: RocksDBOption, TransactionManager, CacheManager, DatabasePath, FileDescriptors, Temp
{startup} feature #38: StorageEngine - depends on: RocksDBEngine, ReplicationTimeout, MMFilesEngine
{startup} feature #39: Database - depends on: MMFilesPersistentIndex, StorageEngine, InitDatabase, Scheduler, CacheManager, Authentication, DatabasePath, EngineSelector
{startup} feature #40: ServerId - depends on: DatabasePath, Database
{startup} feature #41: Replication - depends on: ServerId, StorageEngine, Database
{startup} feature #42: CheckVersion - depends on: Replication, Logger, StorageEngine, Database
{startup} feature #43: MMFilesLogfileManager - depends on: MMFilesEngine, EngineSelector, DatabasePath, Database
{startup} feature #44: MMFilesWalRecovery - depends on: ServerId, Database, MMFilesLogfileManager, MMFilesPersistentIndex, Scheduler
{startup} feature #45: RocksDBRecoveryFinalizer - depends on: ServerId, RocksDBEngine, StorageEngine, Database
{startup} feature #46: TraverserEngineRegistry - depends on: MMFilesLogfileManager, Database, DatabasePath
{startup} feature #47: V8Platform - depends on: Logger
{startup} feature #48: V8Dealer - depends on: MMFilesWalRecovery, Scheduler, V8Platform, Random, WorkMonitor, Action, Authentication, CheckVersion, Temp, Database
{startup} feature #49: Cluster - depends on: V8Dealer, Database, Authentication, Scheduler, CacheManager, Logger, WorkMonitor
{startup} feature #50: FoxxQueues - depends on: V8Platform
{startup} feature #51: QueryRegistry - depends on: MMFilesLogfileManager, Cluster, Database, DatabasePath
{startup} feature #52: Aql - depends on: WorkMonitor, MMFilesLogfileManager, V8Platform, Database, Cluster, Scheduler, CacheManager, QueryRegistry
{startup} feature #53: AQLFunctions - depends on: Aql, EngineSelector
{startup} feature #54: OptimizerRules - depends on: Cluster, Aql, EngineSelector
{startup} feature #55: Statistics - depends on: Aql, Logger
{startup} feature #56: Upgrade - depends on: Aql, CheckVersion, Database, MMFilesWalRecovery, Cluster, V8Dealer
{startup} feature #57: Server - depends on: WorkMonitor, Upgrade, Authentication, MMFilesWalRecovery, V8Dealer, Statistics, Cluster, Temp, Database, Scheduler
{startup} feature #58: Endpoint - depends on: Server, Ssl
{startup} feature #59: Agency (disabled) - depends on: Server, Random, MMFilesWalRecovery, Cluster, Database, Endpoint, Scheduler, QueryRegistry
{startup} feature #60: GeneralServer - depends on: Agency, CheckVersion, Database, Server, Authentication, Endpoint, FoxxQueues, Random, Upgrade, MMFilesLogfileManager, Scheduler
{startup} feature #61: Bootstrap - depends on: V8Dealer, Cluster, CheckVersion, Database, Server, Authentication, Endpoint, FoxxQueues, GeneralServer, MMFilesLogfileManager, Scheduler, Upgrade
{startup} feature #62: Console - depends on: Bootstrap, GeneralServer, Server
{startup} feature #63: Pregel - depends on: Server, V8Dealer, Cluster, WorkMonitor, Database, Logger, Endpoint
{startup} feature #64: Script - depends on: Server, GeneralServer, Nonce, Database
{startup} feature #65: UnitTests - depends on: Server, Bootstrap, GeneralServer, Nonce
{startup} feature #66: Shutdown (disabled) - depends on: Script, UnitTests, Logger
{startup} feature #67: ViewTypes - depends on: WorkMonitor
{startup} ApplicationServer::disableDependentFeatures
{startup} ApplicationServer::daemonize
{startup} Daemon::daemonize
{startup} ApplicationServer::prepare
{startup} dropping privileges
{startup} Jemalloc::prepare
{startup} MMFilesCompaction::prepare
{startup} MaxMapCount::prepare
{startup} ShellColors::prepare
{startup} Version::prepare
{startup} WorkMonitor::prepare
{startup} Logger::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Action::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} FileDescriptors::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Greetings::prepare
2021-07-07T05:09:01Z [1] INFO ArangoDB 3.3.8 [linux] 64bit, using jemalloc, build tags/v3.3.8-0-g50ef880b25, VPack 0.1.30, RocksDB 5.6.0, ICU 58.1, V8 5.7.492.77, OpenSSL 1.1.0f 25 May 2017
2021-07-07T05:09:01Z [1] TRACE {startup} LoggerBuffer::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Nonce::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} PageSize::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Privilege::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Random::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Authentication::prepare
2021-07-07T05:09:01Z [1] INFO {authentication} Jwt secret not specified, generating...
2021-07-07T05:09:01Z [1] TRACE {startup} Scheduler::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} CacheManager::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Config::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Daemon::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Environment::prepare
2021-07-07T05:09:01Z [1] INFO detected operating system: Linux version 4.19.167+ (builder@8ed6aaf7e7f5) (Chromium OS 10.0_pre377782_p20200113-r10 clang version 10.0.0 (/var/cache/chromeos-cache/distfiles/host/egit-src/llvm-project 4e8231b5cf0f5f62c7a51a857e29f5be5cb55734)) #1 SMP Tue Feb 2 07:59:55 PST 2021
2021-07-07T05:09:01Z [1] WARNING {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 2048000
2021-07-07T05:09:01Z [1] WARNING {memory} execute 'sudo sysctl -w "vm.max_map_count=2048000"'
2021-07-07T05:09:01Z [1] TRACE {startup} Frontend::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Language::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Ssl::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} SslServer::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Supervisor::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Temp::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} DatabasePath::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} EngineSelector::prepare
2021-07-07T05:09:01Z [1] DEBUG {startup} looking for previously selected engine in file '/var/lib/arangodb3/ENGINE'
2021-07-07T05:09:01Z [1] INFO using storage engine rocksdb
2021-07-07T05:09:01Z [1] TRACE {startup} disabling storage engine mmfiles
2021-07-07T05:09:01Z [1] TRACE {startup} InitDatabase::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Lockfile::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} RocksDBOption::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} MMFilesPersistentIndex::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} ReplicationTimeout::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} TransactionManager::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} RocksDBEngine::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} StorageEngine::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Database::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} ServerId::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Replication::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} CheckVersion::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} MMFilesLogfileManager::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} MMFilesWalRecovery::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} RocksDBRecoveryFinalizer::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} TraverserEngineRegistry::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} V8Platform::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} V8Dealer::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Cluster::prepare
2021-07-07T05:09:01Z [1] INFO {cluster} Starting up with role SINGLE
2021-07-07T05:09:01Z [1] TRACE {startup} FoxxQueues::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} QueryRegistry::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Aql::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} AQLFunctions::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} OptimizerRules::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Statistics::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Upgrade::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Server::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} raising privileges
2021-07-07T05:09:01Z [1] TRACE {startup} Endpoint::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} dropping privileges
2021-07-07T05:09:01Z [1] TRACE {startup} GeneralServer::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Bootstrap::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Console::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Pregel::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} Script::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} UnitTests::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} ViewTypes::prepare
2021-07-07T05:09:01Z [1] TRACE {startup} ApplicationServer::disableDependentFeatures
2021-07-07T05:09:01Z [1] TRACE {startup} turning off feature 'MMFilesCompaction' because it is enabled only in conjunction with disabled feature 'MMFilesEngine'
2021-07-07T05:09:01Z [1] TRACE {startup} turning off feature 'MMFilesPersistentIndex' because it is enabled only in conjunction with disabled feature 'MMFilesEngine'
2021-07-07T05:09:01Z [1] TRACE {startup} turning off feature 'MMFilesLogfileManager' because it is enabled only in conjunction with disabled feature 'MMFilesEngine'
2021-07-07T05:09:01Z [1] TRACE {startup} turning off feature 'MMFilesWalRecovery' because it is enabled only in conjunction with disabled feature 'MMFilesLogfileManager'
2021-07-07T05:09:01Z [1] TRACE {startup} ApplicationServer::start
2021-07-07T05:09:01Z [1] TRACE {startup} Jemalloc::start
2021-07-07T05:09:01Z [1] TRACE {startup} MaxMapCount::start
2021-07-07T05:09:01Z [1] TRACE {startup} ShellColors::start
2021-07-07T05:09:01Z [1] TRACE {startup} Version::start
2021-07-07T05:09:01Z [1] TRACE {startup} WorkMonitor::start
2021-07-07T05:09:01Z [1] TRACE {startup} Logger::start
2021-07-07T05:09:01Z [1] TRACE {startup} Action::start
2021-07-07T05:09:01Z [1] TRACE {startup} FileDescriptors::start
2021-07-07T05:09:01Z [1] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2021-07-07T05:09:01Z [1] TRACE {startup} Greetings::start
2021-07-07T05:09:01Z [1] TRACE {startup} LoggerBuffer::start
2021-07-07T05:09:01Z [1] TRACE {startup} Nonce::start
2021-07-07T05:09:01Z [1] TRACE {startup} PageSize::start
2021-07-07T05:09:01Z [1] TRACE {startup} Privilege::start
2021-07-07T05:09:01Z [1] TRACE {startup} Random::start
2021-07-07T05:09:01Z [1] TRACE {startup} Authentication::start
2021-07-07T05:09:01Z [1] INFO {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2021-07-07T05:09:01Z [1] TRACE {startup} Scheduler::start
2021-07-07T05:09:01Z [1] DEBUG {startup} scheduler has started
2021-07-07T05:09:01Z [1] TRACE {startup} CacheManager::start
2021-07-07T05:09:01Z [1] DEBUG {startup} cache manager has started
2021-07-07T05:09:01Z [1] TRACE {startup} Config::start
2021-07-07T05:09:01Z [1] TRACE {startup} Daemon::start
2021-07-07T05:09:01Z [1] TRACE {startup} Environment::start
2021-07-07T05:09:01Z [1] WARNING {memory} /proc/sys/vm/overcommit_memory is set to '1'. It is recommended to set it to a value of 2
2021-07-07T05:09:01Z [1] WARNING {memory} execute 'sudo bash -c "echo 2 > /proc/sys/vm/overcommit_memory"'
2021-07-07T05:09:01Z [1] TRACE {startup} Frontend::start
2021-07-07T05:09:01Z [1] TRACE {startup} Language::start
2021-07-07T05:09:01Z [1] TRACE {startup} Ssl::start
2021-07-07T05:09:01Z [1] TRACE {startup} SslServer::start
2021-07-07T05:09:01Z [1] TRACE {startup} Supervisor::start
2021-07-07T05:09:01Z [1] TRACE {startup} Temp::start
2021-07-07T05:09:01Z [1] TRACE {startup} DatabasePath::start
2021-07-07T05:09:01Z [1] TRACE {startup} EngineSelector::start
2021-07-07T05:09:01Z [1] TRACE {startup} InitDatabase::start
2021-07-07T05:09:01Z [1] TRACE {startup} Lockfile::start
2021-07-07T05:09:01Z [1] TRACE {startup} RocksDBOption::start
2021-07-07T05:09:01Z [1] TRACE {startup} ReplicationTimeout::start
2021-07-07T05:09:01Z [1] TRACE {startup} TransactionManager::start
2021-07-07T05:09:01Z [1] TRACE {startup} RocksDBEngine::start
2021-07-07T05:09:02Z [1] DEBUG {startup} found existing column families: default, Documents, PrimaryIndex, EdgeIndex, VPackIndex, GeoIndex, FulltextIndex
2021-07-07T05:09:32Z [1] TRACE {startup} getting existing databases
2021-07-07T05:09:32Z [1] TRACE {startup} StorageEngine::start
2021-07-07T05:09:32Z [1] TRACE {startup} Database::start
2021-07-07T05:09:32Z [1] TRACE {startup} getting existing databases
2021-07-07T05:32:14Z [1] TRACE {startup} ServerId::start
2021-07-07T05:32:14Z [1] TRACE {startup} Replication::start
2021-07-07T05:32:14Z [1] TRACE {startup} CheckVersion::start
2021-07-07T05:32:14Z [1] TRACE {startup} RocksDBRecoveryFinalizer::start
2021-07-07T05:32:14Z [1] TRACE {startup} TraverserEngineRegistry::start
2021-07-07T05:32:14Z [1] TRACE {startup} V8Platform::start
2021-07-07T05:32:14Z [1] TRACE {startup} V8Dealer::start
2021-07-07T05:32:14Z [1] TRACE {startup} Cluster::start
2021-07-07T05:32:14Z [1] TRACE {startup} FoxxQueues::start
2021-07-07T05:32:14Z [1] TRACE {startup} QueryRegistry::start
2021-07-07T05:32:14Z [1] TRACE {startup} Aql::start
2021-07-07T05:32:14Z [1] TRACE {startup} AQLFunctions::start
2021-07-07T05:32:14Z [1] TRACE {startup} OptimizerRules::start
2021-07-07T05:32:14Z [1] TRACE {startup} Statistics::start
2021-07-07T05:32:14Z [1] TRACE {startup} Upgrade::start
2021-07-07T05:32:48Z [1] TRACE {startup} Server::start
2021-07-07T05:32:48Z [1] TRACE {startup} server operation mode: SERVER
2021-07-07T05:32:48Z [1] TRACE {startup} Endpoint::start
2021-07-07T05:32:48Z [1] INFO using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
2021-07-07T05:32:48Z [1] TRACE {startup} GeneralServer::start
2021-07-07T05:32:48Z [1] TRACE {startup} Bootstrap::start
2021-07-07T05:32:48Z [1] DEBUG {startup} Running server/server.js
2021-07-07T05:33:58Z [1] WARNING {v8} giving up waiting for unused V8 context after 60.000000 s
2021-07-07T05:35:20Z [1] WARNING {queries} slow query: '\n FOR item IN @value0\n UPSERT {mount: item[0]}\n INSERT item[1]\n REPLACE item[1]\n IN @@value1\n ', bind vars: {"@value1":"_apps","value0":[["/_admin/aardvark",{"id":"/_admin/aardvark","mount":"/_admin/aardvark","options":{"configuration":{},"dependencies":{},"development":false}}],["/_api/foxx",{"id":"/_api/foxx","mount":"/_api/foxx","options":{"configuration":{},"dependencies":{},"development":false}}],["/_api/gharial",{"id":"/_api/gharial","mount":"/_api/gharial","options":{"configuration":{},"dependencies":{},"development":false}}]]}, took: 98.001884 s
2021-07-07T05:36:18Z [1] WARNING {queries} slow query: '\n FOR item IN @value0\n UPSERT {mount: item[0]}\n INSERT item[1]\n REPLACE item[1]\n IN @@value1\n ', bind vars: {"@value1":"_apps","value0":[["/_admin/aardvark",{"id":"/_admin/aardvark","mount":"/_admin/aardvark","options":{"configuration":{},"dependencies":{},"development":false}}],["/_api/foxx",{"id":"/_api/foxx","mount":"/_api/foxx","options":{"configuration":{},"dependencies":{},"development":false}}],["/_api/gharial",{"id":"/_api/gharial","mount":"/_api/gharial","options":{"configuration":{},"dependencies":{},"development":false}}]]}, took: 58.221921 s

[yogeshbidari]

@Simran-B Any update on the issue.