[TSD-86] Create visual architecture of the project

README.md

@@ -23,6 +23,48 @@ Please read the instructions available in [INSTALL.md](INSTALL.md)
 ![system architecture](https://user-images.githubusercontent.com/15665039/40042756-5d92611e-585d-11e8-80b4-72677c451dd1.png)<br/>
 This is a use case diagram. Use case diagrams overview the usage requirements for a system. They are useful for presentations to management and/or project stakeholders, but for actual development, you will find that use cases provide significantly more value because they describe "the meat" of the actual requirements. For more details, please see [here](http://www.agilemodeling.com/artifacts/useCaseDiagram.htm)
 
+## Architecture
+
+The general architecture can be seen here. It's still missing some things, but it's sufficient for most of the test we currently have.
+
+![log classifier architecture](https://user-images.githubusercontent.com/6264437/43259505-bdaedbf2-90d6-11e8-9b24-fbc6226a3a7e.png)
+
+### Layers
+
+A layer is a list of functions grouped together. For example, a database has functions for interacting with - the database! Yaay. The problem is that in most of the cases we need to be able to stub those functions - if we want to test some of the functions that _depend on the database_, then we have no option but to make them something that we can replace in runtime. And to replace them in runtime, we place them in a "layer", a record-of-functions that allows us to replace them easily. Like:
+```
+data IOLayer m = IOLayer
+    { iolAppendFile :: FilePath -> Text -> m ()
+    , iolPrintText  :: Text -> m ()
+    , iolReadFile   :: FilePath -> m Text
+    , iolLogDebug   :: Text -> m ()
+    , iolLogInfo    :: Text -> m ()
+    }
+```
+
+We can think of a layer like a changeable module system - we export the functions in the data structure, but we can change them in runtime.
+
+### DataSource
+
+The data source is the place we get our data from. Currently, it's fixated on Zendesk. This is an abstraction towards any form of persistant data. We could say that this is a layer, but it actually contains quite a bit more.
+Currently, the Zendesk types themselves are a pretty big part of the `DataSource` module.
+
+### DataLayer
+
+So the data layer is the abstraction layer which can currently be specialized to the @HTTPLayer@ or @DBLayer@. It makes sense to abstract this away, since this is the group of functions that interact with any data in the system.
+
+#### HTTPLayer
+
+Obviously, the direct way we can fetch data is using HTTP JSON requests on the Zendesk API (https://developer.zendesk.com/rest_api/docs/core/introduction). This layer is the layer responsible for that. It contains common functions that allows us to communicate with the Zendesk REST API.
+
+#### HTTPNetworkLayer
+
+This layer is the layer responsible for the low level HTTP communication. The @HTTPLayer@ is the layer that communicates with the Zendesk REST API using this layer.
+
+#### DBLayer
+
+This layer is responsible for caching the results that come from the @HTTPLayer@ and then allows us to fetch data from the database rather then the HTTP REST api which has request limits and takes much longer.
+
 ### Overview
 
 - Many of the Daedalus's issues can be identified by analyzing the log file. The classifier will utilize this by analyzing the log file and map with possible solution and problem which can be provided to the end user.

log-classifier.cabal

@@ -25,8 +25,9 @@ library
                        LogAnalysis.Types
                        Regex
                        Statistics
-                       Util
                        DataSource
+                       Configuration
+                       Util
   other-modules:       Paths_log_classifier
                        DataSource.DB
                        DataSource.Types

src/Configuration.hs

@@ -0,0 +1,41 @@
+-- Here we need to export just the public info.
+module Configuration
+    ( defaultConfig
+    , basicIOLayer
+    ) where
+
+import           Universum
+
+import           DataSource.DB
+import           DataSource.Http
+import           DataSource.Types
+import           HttpLayer (basicHTTPNetworkLayer)
+
+
+-- | The default configuration.
+defaultConfig :: Config
+defaultConfig = Config
+    { cfgAgentId                = 0
+    , cfgZendesk                = "https://iohk.zendesk.com"
+    , cfgToken                  = ""
+    , cfgEmail                  = "daedalus-bug-reports@iohk.io"
+    , cfgAssignTo               = 0
+    , cfgKnowledgebase          = []
+    , cfgNumOfLogsToAnalyze     = 5
+    , cfgIsCommentPublic        = False -- TODO(ks): For now, we need this in CLI.
+    , cfgDataLayer              = basicDataLayer
+    , cfgHTTPNetworkLayer       = basicHTTPNetworkLayer
+    , cfgIOLayer                = basicIOLayer
+    , cfgDBLayer                = connDBLayer
+    }
+
+-- | The @IO@ layer.
+basicIOLayer :: (MonadIO m, MonadReader Config m) => IOLayer m
+basicIOLayer = IOLayer
+    { iolAppendFile             = appendFile
+    , iolPrintText              = putTextLn -- iolPrintConsole
+    , iolReadFile               = readFile
+    , iolLogDebug               = putTextLn
+    , iolLogInfo                = putTextLn
+    }
+

src/DataSource.hs

@@ -3,42 +3,8 @@ module DataSource
     ( module DataSource.Types
     , module DataSource.Http
     , module DataSource.DB
-    , defaultConfig
-    , basicIOLayer
     ) where
 
-import           Universum
-
 import           DataSource.DB
 import           DataSource.Http
 import           DataSource.Types
-import           HttpLayer (basicHTTPNetworkLayer)
-
-
--- | The default configuration.
-defaultConfig :: Config
-defaultConfig = Config
-    { cfgAgentId                = 0
-    , cfgZendesk                = "https://iohk.zendesk.com"
-    , cfgToken                  = ""
-    , cfgEmail                  = "daedalus-bug-reports@iohk.io"
-    , cfgAssignTo               = 0
-    , cfgKnowledgebase          = []
-    , cfgNumOfLogsToAnalyze     = 5
-    , cfgIsCommentPublic        = False -- TODO(ks): For now, we need this in CLI.
-    , cfgZendeskLayer           = basicZendeskLayer
-    , cfgHTTPNetworkLayer       = basicHTTPNetworkLayer
-    , cfgIOLayer                = basicIOLayer
-    , cfgDBLayer                = connDBLayer
-    }
-
--- | The @IO@ layer.
-basicIOLayer :: (MonadIO m, MonadReader Config m) => IOLayer m
-basicIOLayer = IOLayer
-    { iolAppendFile             = appendFile
-    , iolPrintText              = putTextLn -- iolPrintConsole
-    , iolReadFile               = readFile
-    , iolLogDebug               = putTextLn
-    , iolLogInfo                = putTextLn
-    }
-

src/DataSource/DB.hs

@@ -11,8 +11,8 @@ module DataSource.DB
     -- * Empty layer
     , emptyDBLayer
     -- * Single connection
-    , connZendeskLayer
-    , connPoolZendeskLayer
+    , connDataLayer
+    , connPoolDataLayer
     -- * Connection pool
     , connDBLayer
     , connPoolDBLayer
@@ -34,13 +34,13 @@ import           Database.SQLite.Simple.Internal (Connection, Field (..))
 import           Database.SQLite.Simple.Ok (Ok (..))
 import           Database.SQLite.Simple.ToField (ToField (..))
 
-import           DataSource.Http (basicZendeskLayer)
+import           DataSource.Http (basicDataLayer)
 import           DataSource.Types (Attachment (..), AttachmentContent (..), AttachmentId (..),
                                    Comment (..), CommentBody (..), CommentId (..), Config,
                                    DBLayer (..), TicketField (..), TicketFieldId (..),
                                    TicketFieldValue (..), TicketId (..), TicketInfo (..),
                                    TicketStatus (..), TicketTags (..), TicketURL (..), UserId (..),
-                                   ZendeskLayer (..))
+                                   DataLayer (..))
 
 ------------------------------------------------------------
 -- Single connection, simple
@@ -113,18 +113,18 @@ emptyDBLayer = DBLayer
 
 -- | The simple connection Zendesk layer. Used for database querying.
 -- We need to sync occasionaly.
-connZendeskLayer :: forall m. (MonadIO m, MonadReader Config m) => ZendeskLayer m
-connZendeskLayer = ZendeskLayer
+connDataLayer :: forall m. (MonadIO m, MonadReader Config m) => DataLayer m
+connDataLayer = DataLayer
     { zlGetTicketInfo           = \tId -> withProdDatabase $ \conn -> getTicketInfoByTicketId conn tId
-    , zlListDeletedTickets      = zlListDeletedTickets basicZendeskLayer
+    , zlListDeletedTickets      = zlListDeletedTickets basicDataLayer
     , zlListAssignedTickets     = \uId -> withProdDatabase $ \conn -> getAllAssignedTicketsByUser conn uId
     , zlListRequestedTickets    = \uId -> withProdDatabase $ \conn -> getAllRequestedTicketsByUser conn uId
     , zlListUnassignedTickets   =         withProdDatabase getAllUnassignedTicketsByUser
-    , zlListAdminAgents         = zlListAdminAgents basicZendeskLayer
+    , zlListAdminAgents         = zlListAdminAgents basicDataLayer
     , zlGetAttachment           = \att -> withProdDatabase $ \conn -> DataSource.DB.getAttachmentContent conn att
     , zlGetTicketComments       = \tId -> withProdDatabase $ \conn -> getTicketComments conn tId
-    , zlPostTicketComment       = zlPostTicketComment basicZendeskLayer
-    , zlExportTickets           = zlExportTickets basicZendeskLayer
+    , zlPostTicketComment       = zlPostTicketComment basicDataLayer
+    , zlExportTickets           = zlExportTickets basicDataLayer
     }
 
 
@@ -147,18 +147,18 @@ connDBLayer = DBLayer
 
 -- | The connection pooled Zendesk layer. Used for database querying.
 -- We need to sync occasionaly.
-connPoolZendeskLayer :: forall m. (MonadBaseControl IO m, MonadIO m, MonadReader Config m) => DBConnPool -> ZendeskLayer m
-connPoolZendeskLayer connPool = ZendeskLayer
+connPoolDataLayer :: forall m. (MonadBaseControl IO m, MonadIO m, MonadReader Config m) => DBConnPool -> DataLayer m
+connPoolDataLayer connPool = DataLayer
     { zlGetTicketInfo           = \tId -> withConnPool connPool $ \conn -> getTicketInfoByTicketId conn tId
-    , zlListDeletedTickets      = zlListDeletedTickets basicZendeskLayer
+    , zlListDeletedTickets      = zlListDeletedTickets basicDataLayer
     , zlListAssignedTickets     = \uId -> withConnPool connPool $ \conn -> getAllAssignedTicketsByUser conn uId
     , zlListRequestedTickets    = \uId -> withConnPool connPool $ \conn -> getAllRequestedTicketsByUser conn uId
     , zlListUnassignedTickets   =         withConnPool connPool getAllUnassignedTicketsByUser
-    , zlListAdminAgents         = zlListAdminAgents basicZendeskLayer
+    , zlListAdminAgents         = zlListAdminAgents basicDataLayer
     , zlGetAttachment           = \att -> withConnPool connPool $ \conn -> DataSource.DB.getAttachmentContent conn att
     , zlGetTicketComments       = \tId -> withConnPool connPool $ \conn -> getTicketComments conn tId
-    , zlPostTicketComment       = zlPostTicketComment basicZendeskLayer
-    , zlExportTickets           = zlExportTickets basicZendeskLayer
+    , zlPostTicketComment       = zlPostTicketComment basicDataLayer
+    , zlExportTickets           = zlExportTickets basicDataLayer
     }
 
 

src/DataSource/Http.hs

@@ -3,8 +3,8 @@
 {-# LANGUAGE ScopedTypeVariables #-}
 
 module DataSource.Http
-    ( basicZendeskLayer
-    , emptyZendeskLayer
+    ( basicDataLayer
+    , emptyDataLayer
     , createResponseTicket
     ) where
 
@@ -25,7 +25,7 @@ import           DataSource.Types (Attachment (..), AttachmentContent (..), Comm
                                    DeletedTicket (..), ExportFromTime (..), FromPageResultList (..),
                                    PageResultList (..), Ticket (..), TicketId (..), TicketInfo (..),
                                    TicketTag (..), TicketTags (..), User, UserId (..),
-                                   ZendeskAPIUrl (..), ZendeskLayer (..), ZendeskResponse (..),
+                                   ZendeskAPIUrl (..), DataLayer (..), ZendeskResponse (..),
                                    asksHTTPNetworkLayer, parseComments, renderTicketStatus, showURL)
 
 -- ./mitmproxy --mode reverse:https://iohk.zendesk.com -p 4001
@@ -39,8 +39,8 @@ import           DataSource.Types (Attachment (..), AttachmentContent (..), Comm
 --   - get returns a single result (wrapped in @Maybe@)
 --   - list returns multiple results
 --   - post submits a result (maybe PUT?!)
-basicZendeskLayer :: (MonadIO m, MonadReader Config m) => ZendeskLayer m
-basicZendeskLayer = ZendeskLayer
+basicDataLayer :: (MonadIO m, MonadReader Config m) => DataLayer m
+basicDataLayer = DataLayer
     { zlGetTicketInfo           = getTicketInfo
     , zlListDeletedTickets      = listDeletedTickets
     , zlListRequestedTickets    = listRequestedTickets
@@ -54,8 +54,8 @@ basicZendeskLayer = ZendeskLayer
     }
 
 -- | The non-implemented Zendesk layer.
-emptyZendeskLayer :: forall m. (Monad m) => ZendeskLayer m
-emptyZendeskLayer = ZendeskLayer
+emptyDataLayer :: forall m. (Monad m) => DataLayer m
+emptyDataLayer = DataLayer
     { zlGetTicketInfo           = \_     -> error "Not implemented zlGetTicketInfo!"
     , zlListDeletedTickets      =           pure []
     , zlListRequestedTickets    = \_     -> error "Not implemented zlListRequestedTickets!"

src/DataSource/Types.hs

@@ -40,14 +40,14 @@ module DataSource.Types
     -- * General configuration
     , App
     , Config (..)
-    , ZendeskLayer (..)
+    , DataLayer (..)
     , HTTPNetworkLayer (..)
     , IOLayer (..)
     , DBLayer (..)
     , knowledgebasePath
     , tokenPath
     , assignToPath
-    , asksZendeskLayer
+    , asksDataLayer
     , asksHTTPNetworkLayer
     , asksIOLayer
     , asksDBLayer
@@ -115,7 +115,7 @@ data Config = Config
     -- ^ Number of files classifier will analyze
     , cfgIsCommentPublic    :: !Bool
     -- ^ If the comment is public or not, for a test run we use an internal comment.
-    , cfgZendeskLayer       :: !(ZendeskLayer App)
+    , cfgDataLayer          :: !(DataLayer App)
     -- ^ The Zendesk API layer. We will ideally move this into a
     -- separate configuration containing all the layer (yes, there a couple of them).
     , cfgIOLayer            :: !(IOLayer App)
@@ -129,14 +129,14 @@ data Config = Config
     }
 
 
--- | Utility function for getting a function of the @ZendeskLayer@.
-asksZendeskLayer
+-- | Utility function for getting a function of the @DataLayer@.
+asksDataLayer
     :: forall m a. (MonadReader Config m)
-    => (ZendeskLayer App -> a)
+    => (DataLayer App -> a)
     -> m a
-asksZendeskLayer getter = do
+asksDataLayer getter = do
     Config{..} <- ask
-    pure $ getter cfgZendeskLayer
+    pure $ getter cfgDataLayer
 
 -- | Utility function for getting a function of the @cfgHTTPNetworkLayer@.
 asksHTTPNetworkLayer
@@ -183,7 +183,7 @@ assignToPath = "./tmp-secrets/assign_to"
 -- | The Zendesk API interface that we want to expose.
 -- We don't want anything to leak out, so we expose only the most relevant information,
 -- anything relating to how it internaly works should NOT be exposed.
-data ZendeskLayer m = ZendeskLayer
+data DataLayer m = DataLayer
     { zlGetTicketInfo         :: TicketId         -> m (Maybe TicketInfo)
     , zlListDeletedTickets    ::                     m [DeletedTicket]
     , zlListRequestedTickets  :: UserId           -> m [TicketInfo]