Merge branch 'master' of https://github.com/maestro-server/webdocs

maestro-server · Oct 24, 2019 · 92637cc · 92637cc
2 parents 829a1da + 8cfcfb4
commit 92637cc
Show file tree

Hide file tree

Showing 26 changed files with 530 additions and 38 deletions.
diff --git a/docs/_static/analytics_graphs/ana_analytics.png b/docs/_static/analytics_graphs/ana_analytics.png
diff --git a/docs/_static/analytics_graphs/ana_chess.png b/docs/_static/analytics_graphs/ana_chess.png
diff --git a/docs/_static/analytics_graphs/ana_clear.png b/docs/_static/analytics_graphs/ana_clear.png
diff --git a/docs/_static/analytics_graphs/ana_enri.jpg b/docs/_static/analytics_graphs/ana_enri.jpg
diff --git a/docs/_static/analytics_graphs/ana_mongo.jpg b/docs/_static/analytics_graphs/ana_mongo.jpg
diff --git a/docs/_static/analytics_graphs/ana_recursive.jpg b/docs/_static/analytics_graphs/ana_recursive.jpg
diff --git a/docs/_static/analytics_graphs/ana_rotation.png b/docs/_static/analytics_graphs/ana_rotation.png
diff --git a/docs/_static/analytics_graphs/ana_vertical.png b/docs/_static/analytics_graphs/ana_vertical.png
diff --git a/docs/_static/screen/conn_azure.png b/docs/_static/screen/conn_azure.png
diff --git a/docs/_static/screen/conn_do.png b/docs/_static/screen/conn_do.png
diff --git a/docs/_static/screen/conn_do_spaces.png b/docs/_static/screen/conn_do_spaces.png
diff --git a/docs/_static/screen/do_tokens.png b/docs/_static/screen/do_tokens.png
diff --git a/docs/_static/screen/spaces_tokens.png b/docs/_static/screen/spaces_tokens.png
diff --git a/docs/developer/analyticsgraph.rst b/docs/developer/analyticsgraph.rst
@@ -0,0 +1,118 @@
+Algorithm Graphs Analytics
+==========================
+
+This section will expose how analytics graphs works.
+
+Analytics system follow the steps below.
+
+.. image:: ../_static/screen/analytics_internal.png
+   :alt: Maestro Server - Analytics maestro architecture
+
+----------
+
+Making mongodb graph lookup
+---------------------------
+
+Graph lookup will create a python dict using mongodb graph, and all application entities have a dep field, its a list with app objects [id, name].
+
+.. image:: ../_static/analytics_graphs/ana_mongo.jpg
+
+----------
+
+Creating a networkX graph
+-------------------------
+
+After graph sends to network business, basically transform python object to networkX.
+
+We have a recursive function inside each node on the tree, this order will apply a lot of rules creating a new graph tree with better UI order, like in these moments we handle with conflicts, double dependencies, orders and etc.
+
+.. image:: ../_static/analytics_graphs/ana_recursive.jpg
+   :alt: Maestro Server - Analytics Recursive
+
+Code example showing the recursive function
+
+.. code-block:: python
+
+    def _recursive_draw(self, app, i=0, OHelper=HelperOrderedSuccers):
+        if i > 30:
+            return
+
+        for item in app:
+            if not self._grid.in_index(item):
+                node = self._graph.nodes[item]
+                helper = self.add_pos_grid(node)
+
+                succ = OHelper(helper).get_succers()
+                self._recursive_draw(succ, i + 1)
+
+
+In these graphs we can see all rules applied on graph to create a best dependency tree. 
+Each recursive tick try to figure out certainty pattern, if found apply a transformation. 
+For example The child balance pattern try to find a multiple child in one parent, if happen then create a dummy item on parent layer.
+
+-----
+
+
+Rules
+-----
+
+**Growing node**
+
+	- **When:** If the node have more than one child, growing the node to be equal of the number of child
+	- **Transform:** Set the node size to be equal of number of child
+
+**Child Balance**
+
+	- **When:** If the parent node have more than two child.
+	- **Transform:** Create a dummy item beside to node parent.
+
+**Chess Pawn**
+
+	- **When:** If the app it's an entry point and have parent.
+	- **Transform:** Skipped one column
+
+.. image:: ../_static/analytics_graphs/ana_analytics.png
+   :alt: Maestro Server - Analytics
+
+**Chess horse**
+
+	- **When:** If the node have a top obstacle which other nodes point out to a common dependency.
+	- **Transform:** First push back the dependency to a clear column, and after create a dummy path to the new column.
+
+.. image:: ../_static/analytics_graphs/ana_chess.png
+   :alt: Maestro Server - Analytics chess rules
+
+**Clear rows**
+
+	- **When:** After finish the recursive loop, figure out empty columns.
+	- **Transform:** Delete these column and rebalance the grid.
+
+.. image:: ../_static/analytics_graphs/ana_clear.png
+   :alt: Maestro Server - Analytics clear system
+
+
+----------
+
+Enrichment data
+---------------
+
+Next step is an enrichment layer with getting all server data used by each graph.
+
+The enrichment step received two dataset a json python dict represents a graph tree, a matrix position grid. 
+One worker will get all applications accordingly with graph tree creating a new dataset called servers.
+
+.. image:: ../_static/analytics_graphs/ana_enri.jpg
+   :alt: Maestro Server - Analytics Enrichment
+
+----------
+
+Draw layers
+-----------
+
+Now its draw time, where we have three datasets, one its a graph tree, second its matrix od each position and the last its list of servers.
+
+.. image:: ../_static/analytics_graphs/ana_rotation.png
+   :alt: Maestro Server - Analytics Rotation
+
+.. image:: ../_static/analytics_graphs/ana_vertical.png
+   :alt: Maestro Server - Analytics Vertical
diff --git a/docs/developer/architecture/server_app.rst b/docs/developer/architecture/server_app.rst
@@ -171,5 +171,6 @@ Like (PM2):
  AWS_S3_BUCKET_NAME                  maestroserver              Bucket name                                            
  MAESTRO_UPLOAD_TYPE                 S3 or Local                Upload mode                                 
  LOCAL_DIR                           /public/static/            Where files will be uploaded
+ AWS_ENDPOINT                        ny3.spacesdigitalocean     S3 endpoint  
  PWD                                 $rootDirectory             PWD process
 =================================== ========================== ============================================
diff --git a/docs/developer/index.rst b/docs/developer/index.rst
@@ -7,6 +7,7 @@ This chapter will explain internal concepts about Maestro, its only matter if yo
    :maxdepth: 2
 
    architecture/index
+   analyticsgraph
    api/index
    servdiscovery
    tokens

diff --git a/docs/developer/version.rst b/docs/developer/version.rst
@@ -3,14 +3,6 @@ Versions
 
 Microservices compatible versions
 
-------------
-
-v0.5x - Omega
--------------
-
-================  ===========
-Admin             0.5.x
-================  ===========
 
 ------------
 

diff --git a/docs/installing/index.rst b/docs/installing/index.rst
@@ -611,7 +611,14 @@ Advanced configs
 ----------------
 .. toctree::
    :maxdepth: 2
-
+ 
    smtp
    upload
-   themes
+   themes
+
+Deploy Kubernetes
+-----------------
+.. toctree::
+   :maxdepth: 2
+
+   kubernetes
diff --git a/docs/installing/kubernetes.rst b/docs/installing/kubernetes.rst
@@ -0,0 +1,151 @@
+
+You can use kubernetes to run maestro server, 
+the easy way it's create a gke/eks cluster and run all deployment files found in `k8s deployments <https://github.com/maestro-server/infraascode-maestro/tree/master/kubernetes/>`_,
+
+Creating secrets files
+----------------------
+
+First step it's create 4 secrets files (secrets/)
+
+- mongo_srv.txt
+- mongo_uri.txt
+- smtp.txt
+- storage.txt
+
+And populate accordlingly, running
+
+.. code-block:: bash
+
+   kubectl create secret generic smtp --from-env-file secrets/smtp.txt
+   kubectl create secret generic mongo_srv --from-env-file secrets/mongo_srv.txt
+   kubectl create secret generic mongo_uri --from-env-file secrets/mongo_uri.txt
+   kubectl create secret generic storage --from-env-file secrets/storage.txt
+
+**storage.txt**
+
+.. code-block:: bash
+
+	AWS_ACCESS_KEY_ID=
+	AWS_SECRET_ACCESS_KEY=
+	AWS_DEFAULT_REGION=
+	AWS_S3_BUCKET_NAME=
+
+**mongo_uri.txt**
+
+.. code-block:: bash
+
+	MAESTRO_MONGO_URI=mongo://mongodb:27017
+
+**mongo_srv.txt**
+
+.. code-block:: bash
+
+	MAESTRO_MONGO_URI=mongo+srv://mongodb:27017
+
+**smtp.txt**
+
+.. code-block:: bash
+
+	SMTP_PORT=
+	SMTP_HOST=
+	SMTP_SENDER=
+	SMTP_USERNAME=
+	SMTP_PASSWORD=
+	SMTP_USETSL=
+
+To check if everything it's ok run 
+
+.. code-block:: bash
+
+	> kubectl get secrets
+
+	NAME                  TYPE                                  DATA      AGE
+	mongosrv              Opaque                                1         24d
+	mongouri              Opaque                                1         24d
+	smtp                  Opaque                                6         18d
+	storage               Opaque                                4         17d
+
+--------
+
+Deployment each service
+-----------------------
+
+.. code-block:: bash
+
+	source run.sh
+
+Or
+
+Create a third services.
+
+.. code-block:: bash
+
+	kubectl apply -f mongo/
+	kubectl apply -f rabbitmq/
+	kubectl apply -f maildev/
+
+Maestro micro services deployments.
+
+.. code-block:: bash
+
+	kubectl apply -f maestro-websocket/
+	kubectl apply -f maestro-data/
+	kubectl apply -f maestro-discovery/
+	kubectl apply -f maestro-reports/
+	kubectl apply -f maestro-analytics/
+	kubectl apply -f maestro-analytics-front/
+	kubectl apply -f maestro-audit/
+	kubectl apply -f maestro-scheduler/
+	kubectl apply -f maestro-server/
+	kubectl apply -f maestro-client/
+
+---------
+
+Checking deployments
+--------------------
+
+.. code-block:: bash
+
+	> kubectl get deployments
+
+	NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
+	maestro-analytics          1         1         1            1           6d
+	maestro-analytics-front    1         1         1            1           6d
+	maestro-analytics-worker   1         1         1            1           6d
+	maestro-audit              1         1         1            1           6d
+	maestro-data               1         1         1            1           24d
+	maestro-discovery          1         1         1            1           6d
+	maestro-discovery-worker   1         1         1            1           6d
+	maestro-reports            1         1         1            1           6d
+	maestro-reports-worker     1         1         1            1           6d
+	maestro-scheduler          1         1         1            1           6d
+	maestro-scheduler-worker   1         1         1            1           6d
+	maestro-server             2         2         2            2           6d
+	maestro-websocket          1         1         1            1           6d
+	rabbitmq                   1         1         1            1           24d
+
+
+Checking exposed services
+-------------------------
+
+.. code-block:: bash
+
+	> kubectl get svc  
+
+	NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                       AGE
+	external-analytics-front   LoadBalancer   10.XX.252.63    XX.XX.XX.XX      80:30859/TCP                  23d
+	external-server            LoadBalancer   10.XX.245.248   XX.XX.XX.XX      80:31254/TCP                  23d
+	external-websocket         LoadBalancer   10.XX.253.161   XX.XX.XX.XX      8443:30705/TCP,80:31146/TCP   21d
+
+	internal-analytics         ClusterIP      10.XX.240.129   <none>           5020/TCP                      6d
+	internal-analytics-front   ClusterIP      10.XX.243.157   <none>           9999/TCP                      23d
+	internal-audit             ClusterIP      10.XX.243.250   <none>           10900/TCP                     6d
+	internal-data              ClusterIP      10.XX.244.111   <none>           5010/TCP                      24d
+	internal-discovery         ClusterIP      10.XX.240.202   <none>           5000/TCP                      6d
+	internal-rabbit            ClusterIP      10.XX.243.117   <none>           5672/TCP,15672/TCP            24d
+	internal-reports           ClusterIP      10.XX.241.218   <none>           5005/TCP                      6d
+	internal-websocket         ClusterIP      10.XX.241.159   <none>           8000/TCP                      21d
+
+.. Note::
+
+    Must have 3 public endpoint, it's server app, analytics front and websocket system.
diff --git a/docs/installing/upload.rst b/docs/installing/upload.rst
@@ -50,7 +50,8 @@ Env variables
   AWS_ACCESS_KEY_ID       XXXXXXXXXX                      
   AWS_SECRET_ACCESS_KEY   XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  
   AWS_DEFAULT_REGION      us-east-1                       
-  AWS_S3_BUCKET_NAME      maestroserver                   
+  AWS_S3_BUCKET_NAME      maestroserver
+  AWS_ENDPOINT            S3 endpoint       
  ======================= ================================ 
 
  .. code-block:: yaml
@@ -75,6 +76,26 @@ Env variables
 
     Need to be adjusted client-app appoint new local file
 
+Digital Ocean Spaces
+--------------------
+
+You can use Digital ocean space
+
+Env variables
+
+======================= ================================ 
+UPLOAD_TYPE             S3 
+AWS_ACCESS_KEY_ID       XXXXXXXXXX                      
+AWS_SECRET_ACCESS_KEY   XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  
+AWS_DEFAULT_REGION      us-east-1                       
+AWS_S3_BUCKET_NAME      maestroserver
+AWS_ENDPOINT            S3 endpoint       
+======================= ================================ 
+
+Endpoint can be ny3.spacesdigitalocean
+Access and secret it's create on spaces dashboard.
+AWS_DEFAULT_REGION can be ny3
+
 -------
 
 Services

diff --git a/docs/quickstart.rst b/docs/quickstart.rst
@@ -230,6 +230,8 @@ We recommend to use docker, if you like to see demo version, copy and execute do
     Remember to config API_URL and STATIC_URL on client app with ip/dns of your server.
 
 
+-----------------
+
 Vagrant
 -------
 
@@ -247,4 +249,16 @@ We have VagrantFile, its good for visualization (demo) or the best way to create
 
     If your necessity is, HA, critical situation, go in `Ha session`__.
 
-    __ installing/production.html
+    __ installing/production.html
+
+-----------------
+
+Kubernetes
+-----------------
+
+You can use kubernetes to run maestro server.
+
+.. toctree::
+   :maxdepth: 1
+
+   ../installing/kubernetes