core: servers summary api (#4319)

* core: servers summary api

* rework

* server known types

* set stats file path

* rename

* local simplexmq

* update

* rfc

* update

* update

* get servers

* compile summary

* remove sort

* rename

* rename, refactor

* refactor attempt

* refactor attempt 2

* refactor

* fix

* fix2

* remove space

* refactor xftp

* update

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>

cabal.project

@@ -12,7 +12,7 @@ constraints: zip +disable-bzip2 +disable-zstd
 source-repository-package
     type: git
     location: https://github.com/simplex-chat/simplexmq.git
-    tag: 8a3b72458f917e9867f4e3640dda0fa1827ff6cf
+    tag: c7886926870e97fa592d51fa36a2cdec49296388
 
 source-repository-package
     type: git

docs/rfcs/2024-06-17-agent-stats-persistence.md

@@ -0,0 +1,87 @@
+# Agent stats persistence
+
+## Problem
+
+State/state tracked in agent are lost on app restart, which makes it difficult to debug user bugs.
+
+## Solution
+
+Persist stats between sessions.
+
+App terminal signals may vary per platform / be absent (?) -> persist stats periodically.
+
+Stats would have `<userId, server>` key, so we don't want to store them in a plaintext file to not leak used servers locally -> persist in encrypted db.
+
+There's couple of orthogonal design decision to be made:
+- persist in chat or in agent db
+  - pros for chat:
+    - possibly less contention for db than agent
+  - pros for agent:
+    - no unnecessary back and forth, especially if agent starts accumulating from past sessions and has to be parameterized with past stats (see below)
+- agent to start accumulating from past sessions stats, or keep past separately and only accumulate for current session from zeros
+  - pros for accumulating from past sessions:
+    - easier to maintain stats - e.g. user deletion has to remove keys, which is more convoluted if past stats are not stored in memory
+    - simpler UI - overall stats, no differentiation for past/current session (or less logic in backend preparing presentation data)
+  - pros for accumulating from zeros:
+    - simpler start logic - no need to restore stats from agent db / pass initial stats from chat db
+    - can differentiate between past sessions and current session stats in UI
+
+### Option 1 - Persist in chat db, agent to track only current session
+
+- Chat stores stats in such table:
+
+```sql
+CREATE TABLE agent_stats(
+  agent_stats_id INTEGER PRIMARY KEY, -- dummy id, there will only be one record
+  past_stats TEXT, -- accumulated from previous sessions
+  session_stats TEXT, -- current session
+  past_started_at TEXT NOT NULL DEFAULT(datetime('now')), -- starting point of tracking stats, reset on stats reset
+  session_started_at TEXT NOT NULL DEFAULT(datetime('now')), -- starting point of current session
+  session_updated_at TEXT NOT NULL DEFAULT(datetime('now')) -- last update of current session stats (periodic, frequent updates)
+);
+```
+
+- Chat periodically calls getAgentServersStats api and updates `session_stats`.
+  - interval? should be short to not lose too much data, 5-30 seconds?
+- On start `session_stats` are accumulated into `past_stats` and set to null.
+- On user deletion, agent updates current session stats in memory (removes keys), chat has to do same for both stats fields in db.
+  - other cases where stats have to be manipulated in similar way?
+
+### Option 2 - Persist in chat db, agent to accumulate stats from past sessions
+
+- Table is only used for persistence of overall stats:
+
+```sql
+CREATE TABLE agent_stats(
+  agent_stats_id INTEGER PRIMARY KEY, -- dummy id, there will only be one record
+  agent_stats TEXT, -- overall stats - past and session
+  started_tracking_at TEXT NOT NULL DEFAULT(datetime('now')), -- starting point of tracking stats, reset on stats reset
+  updated_at TEXT NOT NULL DEFAULT(datetime('now'))
+);
+```
+
+- Chat to parameterize creation of agent client with initial stats.
+
+### Option 3 - Persist in agent db, agent to differentiate past stats and session stats
+
+- Table in agent db similar to option 1.
+- Agent is responsible for periodic updates in session, as well as accumulating into "past" and resetting session stats on start.
+- Agent only communicates stats to chat on request.
+- On user deletion agent is fully responsible for maintaining both in-memory session stats, and updating db records.
+
+### Option 4 - Persist in agent db, agent to accumulate stats from past sessions
+
+- Table in agent db similar to option 2.
+- On start agent restores initial stats into memory by itself.
+- Since all stats are in memory, on user deletion it's enough to update in memory without updating db.
+  - there is a race possible where agent crashes after updating stats (removing user keys) in memory before database stats have been overwritten by a periodic update, so it may be better to immediately overwrite and not wait for periodic update.
+  - still at least there's at least no additional logic to update past stats.
+
+### Other considerations
+
+Why is it important to timely remove user keys from past stats?
+- stats not being saved for past users:
+  - important both privacy-wise and to not cause confusion when showing "All" stats (e.g. user summing up across users stats would have smaller total than total stats).
+  - to avoid accidentally mixing up with newer users.
+    - though we do have an AUTOINCREMENT user_id in agent so probably it wouldn't be a problem.
+- on the other hand maybe we don't want to "forget" stats on user deletion so that stats would reflect networking more accurately?

scripts/nix/sha256map.nix

@@ -1,5 +1,5 @@
 {
-  "https://github.com/simplex-chat/simplexmq.git"."8a3b72458f917e9867f4e3640dda0fa1827ff6cf" = "1mmxdaj563kjmlkacxdnq62n6mzw9khampzaqghnk6iiwzdig0qy";
+  "https://github.com/simplex-chat/simplexmq.git"."c7886926870e97fa592d51fa36a2cdec49296388" = "1r3nibcgw3whl0q3ssyr1606x4ilqphhzqyihi3aw4nw5fmz226h";
   "https://github.com/simplex-chat/hs-socks.git"."a30cc7a79a08d8108316094f8f2f82a0c5e1ac51" = "0yasvnr7g91k76mjkamvzab2kvlb1g5pspjyjn2fr6v83swjhj38";
   "https://github.com/simplex-chat/direct-sqlcipher.git"."f814ee68b16a9447fbb467ccc8f29bdd3546bfd9" = "1ql13f4kfwkbaq7nygkxgw84213i0zm7c1a8hwvramayxl38dq5d";
   "https://github.com/simplex-chat/sqlcipher-simple.git"."a46bd361a19376c5211f1058908fc0ae6bf42446" = "1z0r78d8f0812kxbgsm735qf6xx8lvaz27k1a0b4a2m0sshpd5gl";

simplex-chat.cabal

@@ -160,6 +160,7 @@ library
       Simplex.Chat.Remote.RevHTTP
       Simplex.Chat.Remote.Transport
       Simplex.Chat.Remote.Types
+      Simplex.Chat.Stats
       Simplex.Chat.Store
       Simplex.Chat.Store.AppSettings
       Simplex.Chat.Store.Connections

src/Simplex/Chat.hs

@@ -20,6 +20,7 @@ import Control.Applicative (optional, (<|>))
 import Control.Concurrent.STM (retry)
 import Control.Logger.Simple
 import Control.Monad
+import Simplex.Chat.Stats
 import Control.Monad.Except
 import Control.Monad.IO.Unlift
 import Control.Monad.Reader
@@ -84,7 +85,6 @@ import Simplex.Chat.Store.Shared
 import Simplex.Chat.Types
 import Simplex.Chat.Types.Preferences
 import Simplex.Chat.Types.Shared
-import Simplex.Chat.Types.Util
 import Simplex.Chat.Util (encryptFile, liftIOEither, shuffle)
 import qualified Simplex.Chat.Util as U
 import Simplex.FileTransfer.Client.Main (maxFileSize, maxFileSizeHard)
@@ -113,7 +113,7 @@ import qualified Simplex.Messaging.Crypto.Ratchet as CR
 import Simplex.Messaging.Encoding
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (base64P)
-import Simplex.Messaging.Protocol (AProtoServerWithAuth (..), AProtocolType (..), EntityId, ErrorType (..), MsgBody, MsgFlags (..), NtfServer, ProtoServerWithAuth (..), ProtocolTypeI, SProtocolType (..), SubscriptionMode (..), UserProtocol, XFTPServer, userProtocol)
+import Simplex.Messaging.Protocol (AProtoServerWithAuth (..), AProtocolType (..), EntityId, ErrorType (..), MsgBody, MsgFlags (..), NtfServer, ProtoServerWithAuth (..), ProtocolTypeI, SProtocolType (..), SubscriptionMode (..), UserProtocol, XFTPServer, userProtocol, ProtocolServer)
 import qualified Simplex.Messaging.Protocol as SMP
 import Simplex.Messaging.ServiceScheme (ServiceScheme (..))
 import qualified Simplex.Messaging.TMap as TM
@@ -369,7 +369,7 @@ activeAgentServers ChatConfig {defaultServers} p =
   fromMaybe (cfgServers p defaultServers)
     . nonEmpty
     . map (\ServerCfg {server} -> server)
-    . filter (\ServerCfg {enabled} -> enabled)
+    . filter (\ServerCfg {enabled} -> enabled == SEEnabled)
 
 cfgServers :: UserProtocol p => SProtocolType p -> (DefaultAgentServers -> NonEmpty (ProtoServerWithAuth p))
 cfgServers p DefaultAgentServers {smp, xftp} = case p of
@@ -1315,7 +1315,7 @@ processChatCommand' vr = \case
         servers' = fromMaybe (L.map toServerCfg defServers) $ nonEmpty servers
     pure $ CRUserProtoServers user $ AUPS $ UserProtoServers p servers' defServers
     where
-      toServerCfg server = ServerCfg {server, preset = True, tested = Nothing, enabled = True}
+      toServerCfg server = ServerCfg {server, preset = True, tested = Nothing, enabled = SEEnabled}
   GetUserProtoServers aProtocol -> withUser $ \User {userId} ->
     processChatCommand $ APIGetUserProtoServers userId aProtocol
   APISetUserProtoServers userId (APSC p (ProtoServersConfig servers)) -> withUserId userId $ \user -> withServerProtocol p $ do
@@ -2253,6 +2253,21 @@ processChatCommand' vr = \case
         CLUserContact ucId -> "UserContact " <> show ucId
         CLFile fId -> "File " <> show fId
   DebugEvent event -> toView event >> ok_
+  GetAgentServersSummary userId -> withUserId userId $ \user -> do
+    agentServersSummary <- lift $ withAgent' getAgentServersSummary
+    users <- withStore' getUsers
+    smpServers <- getUserServers user SPSMP
+    xftpServers <- getUserServers user SPXFTP
+    let presentedServersSummary = toPresentedServersSummary agentServersSummary users user smpServers xftpServers
+    pure $ CRAgentServersSummary user presentedServersSummary
+    where
+      getUserServers :: forall p. (ProtocolTypeI p, UserProtocol p) => User -> SProtocolType p -> CM [ProtocolServer p]
+      getUserServers users protocol = do
+        ChatConfig {defaultServers} <- asks config
+        let defServers = cfgServers protocol defaultServers
+        servers <- map (\ServerCfg {server} -> server) <$> withStore' (`getProtocolServers` users)
+        let srvs = if null servers then L.toList defServers else servers
+        pure $ map protoServer srvs
   GetAgentWorkers -> lift $ CRAgentWorkersSummary <$> withAgent' getAgentWorkersSummary
   GetAgentWorkersDetails -> lift $ CRAgentWorkersDetails <$> withAgent' getAgentWorkersDetails
   GetAgentStats -> lift $ CRAgentStats . map stat <$> withAgent' getAgentStats
@@ -7611,6 +7626,7 @@ chatCommandP =
       ("/version" <|> "/v") $> ShowVersion,
       "/debug locks" $> DebugLocks,
       "/debug event " *> (DebugEvent <$> jsonP),
+      "/get servers summary " *> (GetAgentServersSummary <$> A.decimal),
       "/get stats" $> GetAgentStats,
       "/reset stats" $> ResetAgentStats,
       "/get subs" $> GetAgentSubs,
@@ -7755,7 +7771,7 @@ chatCommandP =
         (Just <$> (AutoAccept <$> (" incognito=" *> onOffP <|> pure False) <*> optional (A.space *> msgContentP)))
         (pure Nothing)
     srvCfgP = strP >>= \case AProtocolType p -> APSC p <$> (A.space *> jsonP)
-    toServerCfg server = ServerCfg {server, preset = False, tested = Nothing, enabled = True}
+    toServerCfg server = ServerCfg {server, preset = False, tested = Nothing, enabled = SEEnabled}
     rcCtrlAddressP = RCCtrlAddress <$> ("addr=" *> strP) <*> (" iface=" *> (jsonP <|> text1P))
     text1P = safeDecodeUtf8 <$> A.takeTill (== ' ')
     char_ = optional . A.char

src/Simplex/Chat/Call.hs

@@ -21,10 +21,10 @@ import Data.Time.Clock (UTCTime)
 import Database.SQLite.Simple.FromField (FromField (..))
 import Database.SQLite.Simple.ToField (ToField (..))
 import Simplex.Chat.Types (Contact, ContactId, User)
-import Simplex.Chat.Types.Util (decodeJSON, encodeJSON)
 import qualified Simplex.Messaging.Crypto as C
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, enumJSON, fromTextField_, fstToLower, singleFieldJSON)
+import Simplex.Messaging.Util (decodeJSON, encodeJSON)
 
 data Call = Call
   { contactId :: ContactId,

src/Simplex/Chat/Controller.hs

@@ -60,6 +60,7 @@ import Simplex.Chat.Messages.CIContent
 import Simplex.Chat.Protocol
 import Simplex.Chat.Remote.AppVersion
 import Simplex.Chat.Remote.Types
+import Simplex.Chat.Stats (PresentedServersSummary)
 import Simplex.Chat.Store (AutoAccept, ChatLockEntity, StoreError (..), UserContactLink, UserMsgReceiptSettings)
 import Simplex.Chat.Types
 import Simplex.Chat.Types.Preferences
@@ -75,7 +76,7 @@ import Simplex.Messaging.Agent.Protocol
 import Simplex.Messaging.Agent.Store.SQLite (MigrationConfirmation, SQLiteStore, UpMigration, withTransaction)
 import Simplex.Messaging.Agent.Store.SQLite.DB (SlowQueryStats (..))
 import qualified Simplex.Messaging.Agent.Store.SQLite.DB as DB
-import Simplex.Messaging.Client (SMPProxyMode (..), SMPProxyFallback (..))
+import Simplex.Messaging.Client (SMPProxyFallback (..), SMPProxyMode (..))
 import qualified Simplex.Messaging.Crypto as C
 import Simplex.Messaging.Crypto.File (CryptoFile (..))
 import qualified Simplex.Messaging.Crypto.File as CF
@@ -505,6 +506,7 @@ data ChatCommand
   | ShowVersion
   | DebugLocks
   | DebugEvent ChatResponse
+  | GetAgentServersSummary UserId
   | GetAgentStats
   | ResetAgentStats
   | GetAgentSubs
@@ -756,6 +758,7 @@ data ChatResponse
   | CRSQLResult {rows :: [Text]}
   | CRSlowSQLQueries {chatQueries :: [SlowSQLQuery], agentQueries :: [SlowSQLQuery]}
   | CRDebugLocks {chatLockName :: Maybe String, chatEntityLocks :: Map String String, agentLocks :: AgentLocks}
+  | CRAgentServersSummary {user :: User, serversSummary :: PresentedServersSummary}
   | CRAgentStats {agentStats :: [[String]]}
   | CRAgentWorkersDetails {agentWorkersDetails :: AgentWorkersDetails}
   | CRAgentWorkersSummary {agentWorkersSummary :: AgentWorkersSummary}

src/Simplex/Chat/Markdown.hs

@@ -29,13 +29,12 @@ import Data.Text (Text)
 import qualified Data.Text as T
 import Data.Text.Encoding (encodeUtf8)
 import Simplex.Chat.Types
-import Simplex.Chat.Types.Util
 import Simplex.Messaging.Agent.Protocol (AConnectionRequestUri (..), ConnReqUriData (..), ConnectionRequestUri (..), SMPQueue (..))
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, enumJSON, fstToLower, sumTypeJSON)
 import Simplex.Messaging.Protocol (ProtocolServer (..))
 import Simplex.Messaging.ServiceScheme (ServiceScheme (..))
-import Simplex.Messaging.Util (safeDecodeUtf8)
+import Simplex.Messaging.Util (decodeJSON, safeDecodeUtf8)
 import System.Console.ANSI.Types
 import qualified Text.Email.Validate as Email
 
@@ -146,7 +145,7 @@ parseMarkdown s = fromRight (unmarked s) $ A.parseOnly (markdownP <* A.endOfInpu
 
 isSimplexLink :: Format -> Bool
 isSimplexLink = \case
-  SimplexLink {} -> True;
+  SimplexLink {} -> True
   _ -> False
 
 markdownP :: Parser Markdown

src/Simplex/Chat/Messages/CIContent.hs

@@ -29,12 +29,11 @@ import Simplex.Chat.Protocol
 import Simplex.Chat.Types
 import Simplex.Chat.Types.Preferences
 import Simplex.Chat.Types.Shared
-import Simplex.Chat.Types.Util
 import Simplex.Messaging.Agent.Protocol (MsgErrorType (..), RatchetSyncState (..), SwitchPhase (..))
-import Simplex.Messaging.Crypto.Ratchet (PQEncryption, pattern PQEncOn, pattern PQEncOff)
+import Simplex.Messaging.Crypto.Ratchet (PQEncryption, pattern PQEncOff, pattern PQEncOn)
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, enumJSON, fstToLower, singleFieldJSON, sumTypeJSON)
-import Simplex.Messaging.Util (safeDecodeUtf8, tshow, (<$?>))
+import Simplex.Messaging.Util (encodeJSON, safeDecodeUtf8, tshow, (<$?>))
 
 data MsgDirection = MDRcv | MDSnd
   deriving (Eq, Show)

src/Simplex/Chat/Protocol.hs

@@ -46,14 +46,13 @@ import Database.SQLite.Simple.ToField (ToField (..))
 import Simplex.Chat.Call
 import Simplex.Chat.Types
 import Simplex.Chat.Types.Shared
-import Simplex.Chat.Types.Util
 import Simplex.Messaging.Agent.Protocol (VersionSMPA, pqdrSMPAgentVersion)
 import Simplex.Messaging.Compression (Compressed, compress1, decompress1)
 import Simplex.Messaging.Encoding
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, fromTextField_, fstToLower, parseAll, sumTypeJSON, taggedObjectJSON)
 import Simplex.Messaging.Protocol (MsgBody)
-import Simplex.Messaging.Util (eitherToMaybe, safeDecodeUtf8, (<$?>))
+import Simplex.Messaging.Util (decodeJSON, eitherToMaybe, encodeJSON, safeDecodeUtf8, (<$?>))
 import Simplex.Messaging.Version hiding (version)
 
 -- Chat version history: