Merge pull request #35 from muralibasani/5.0.0

Kafkawize pro changes

3.1/docs/conf.py

@@ -51,9 +51,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '4.5.1'
+version = '5.0.0'
 # The full version, including alpha/beta/rc tags.
-release = '4.5.1'
+release = '5.0.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

3.1/docs/docker/docker.rst

@@ -1,7 +1,7 @@
 On Docker/Azure Cloud
 =====================
 
-Kafkawize can be run with Docker from version 4.5.1 You can download the docker images from https://hub.docker.com/u/kafkawize
+Kafkawize can be run with Docker from version 5.0.0 You can download the docker images from https://hub.docker.com/u/kafkawize
 
 Step 1 (Docker installation)
 ----------------------------
@@ -19,7 +19,7 @@ For Azure, make sure images are correctly defined.
     version: '2'
     services:
       uiapi:
-        image: kafkawize/kw_uiapi:4.5.1
+        image: kafkawize/kw_uiapi:5.0.0
         hostname: localhost
         environment:
           KAFKAWIZE_DB_STORETYPE: rdbms
@@ -28,11 +28,9 @@ For Azure, make sure images are correctly defined.
           KAFKAWIZE_DBSCRIPTS_CREATE_TABLES: "true"
           KAFKAWIZE_DBSCRIPTS_INSERT_BASICDATA: "true"
 
-          # License key settings
-          KAFKAWIZE_LICENSE_KEY: <key-to-be-filled>
+          # Other settings
           KAFKAWIZE_ORG_NAME: MyOrganization
-          KAFKAWIZE_VERSION: 4.5.1
-          KAFKAWIZE_INVALIDKEY_MSG: Invalid License !! Please request from https://kafkawize.com for a license key.
+          KAFKAWIZE_VERSION: 5.0.0
 
           # Database settings
           SPRING_DATASOURCE_URL: jdbc:h2:file:./kafkawizedbos;DB_CLOSE_ON_EXIT=FALSE;DB_CLOSE_DELAY=-1;MODE=MySQL;DATABASE_TO_LOWER=TRUE;
@@ -65,7 +63,7 @@ Create a docker compose file(kafkawize_docker_compose_clusterapi.yml) like below
     version: '2'
     services:
       clusterapi:
-        image: kafkawize/kw_clusterapi:4.5.1
+        image: kafkawize/kw_clusterapi:5.0.0
         environment:
           LOGGING_FILE: kw_clusterapi.log
         ports:
@@ -101,13 +99,7 @@ Credentials
 
 Default Credentials available to access Kafkawize::
 
-    gary/user USER Octopus
-    will/user USER Seahorses
-    john/user USER Starfish
-    cris/user ADMIN Octopus
-    noah/user ADMIN Seahorses
-    alex/user ADMIN Starfish
-    superuser/user SUPERUSER Seahorses
+    superadmin/kwsuperadmin123$$
 
 
 Docker shell

3.1/docs/features/features.rst

@@ -6,35 +6,31 @@ Kafkawize is built with the key features as below::
    -    4 eyed principle – Requesting and Approving topics/acls/schemas
    -    Spring based security
    -    Fully automated
-   -    Browse Acls,  Producers and Consumers
-   -    Synchronize Source of truth with Meta store
+   -    Browse Topics, Acls, Schemas, Connectors,  Producers and Consumers
+   -    Synchronize Source of truth with Meta store, Restore config
    -    Support for an Rdbms or file system as metastore
 
+
+.. raw:: html
+
+    <iframe width="560" height="315" src="https://www.youtube.com/embed/LOqjwARmbBs" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+
+
+
 Security
 --------
-There are two main applications of Kafkawize. UiApi and ClusterApi. There is no security applied on Cluster Api. End points are exposed to anyone who can reach the host., and swagger is enabled too. Users can directly post/get requests to this application.
+There are two main applications of Kafkawize. UiApi and ClusterApi. There is a security applied on Cluster Api. End points are exposed to anyone who can reach the host and has access., and swagger is enabled too. Users can directly post/get requests to this application.
 
 On the UI Api, Spring security is enabled. As the application boots, it loads all the users from database into its memory and would be referred to it.
 
-Security on the Kafka Broker is out of scope for Kafkawize., and it should be handled by you., however for Kafkawize to connect to Kafka broker, Acls/AdminClient properties need to be provided to avoid authorization issues.
+Security on the Kafka Broker is out of scope for Kafkawize., , however for Kafkawize to connect to Kafka broker, Acls/AdminClient properties need to be provided to avoid authorization issues.
 
 Teams and Role Management
 -------------------------
 
-Kafkawize comes with 3 mains roles::
+Kafkawize can be configured with any number of roles and permissions::
 
-    -   USER
-    -   ADMIN
-    -   SUPERUSER
-
-'User' has the rights to request and browse data
-
-'Admin' has the all the rights as USER and approve all requests
-
-'Superuser' has all the rights as ADMIN and synchronize cluster data with metadata. For ex : assign a team to a topic or acl.
-
-A team can have multiple users with the above roles. Depending on the role of a user, he is authorized to perform the relevant actions on the portal.
-User receives an authorization error if he/she performs an action if not authorized to.
+By default 'USER' roles can request and approve topic/acl/schema/connector requests.
 
 Login
 -----
@@ -177,7 +173,7 @@ All the requests from users are audited and can be viewed.
 
 My Topic Requests
 ~~~~~~~~~~~~~~~~~
-Requests (Topics/Acls/Schemas) from their own team can be viewed and deleted.
+Requests (Topics/Acls/Schemas/Connectors) from their own team can be viewed and deleted.
 
 .. image:: _static/images/MyTopicRequests.JPG
     :width: 500px
@@ -203,25 +199,25 @@ All the requests requested and approved can be viewed. Users login/logout are no
 Approve Or Decline
 ------------------
 
-Topic Requests can be approved or declined by Admins or Super users from the same team. After approval of a topic, it is created on the cluster, how ever no acls are assigned to it yet.
+Topic Requests can be approved or declined by users from the same team. After approval of a topic, it is created on the cluster, how ever no acls are assigned to it yet.
 
-Acl Requests can be approved/declined by Admins or Super users who belong to Topic Owner's team. After approval, acls are created on the cluster.
+Acl Requests can be approved/declined by users who belong to Topic Owner's team. After approval, acls are created on the cluster.
 
 Schema Requests can be approved by Admins or Super users from the same team.
 
 Notifications are visible on the top right side of the portal.
 
 Topics
 ~~~~~~
-If a topic is requested by 'user1' from 'Team1', it can be approved/declined by 'user2' from 'Team1' only., provided 'user2' has Admin/Super role.
+If a topic is requested by 'user1' from 'Team1', it can be approved/declined by 'user2' from 'Team1' only., provided 'user2' has same role.
 
 .. image:: _static/images/ApproveTopics.JPG
     :width: 500px
     :align: center
 
 Acls
 ~~~~
-If a acl is requested by 'user1' on topic which is owned by 'Team2', it can be approved/declined by 'user2' from 'Team2' only., provided 'user2' has Admin/Super role.
+If a acl is requested by 'user1' on topic which is owned by 'Team2', it can be approved/declined by 'user2' from 'Team2' only., provided 'user2' has same role.
 
 .. image:: _static/images/ApproveACL.JPG
     :width: 500px
@@ -230,7 +226,7 @@ If a acl is requested by 'user1' on topic which is owned by 'Team2', it can be a
 Avro Schemas
 ~~~~~~~~~~~~
 
-If a schema is requested by 'user1' from 'Team1', it can be approved by 'user2' from 'Team1' only., provided 'user2' has Admin/Super role.
+If a schema is requested by 'user1' from 'Team1', it can be approved by 'user2' from 'Team1' only., provided 'user2' has same role.
 
 Users
 -----
@@ -298,3 +294,165 @@ Logout
 ------
 
 Users can logout after clicking on the logout button on the top right corner. Session will be killed after this action.
+
+Promote Topics
+--------------
+
+If a topic exists in Dev environment, it can be requested for promotion to the next higher environment (TST). If topic exists in Dev and Tst environments,
+promotion can be requested for Acceptance environment, with the same topic name. This way it restricts users to create topics in adhoc way
+and at the same time maintains data integrity. This feature provides an easy way to request for topics in higher environments, keeping
+partitions, replication factor parameters specific to those environments. This deletion request can be triggered only by Topic owner teams.
+Please note, it follows the same request/approval process.
+
+Delete Topics
+-------------
+
+When a topic is no more needed,it is best to get it deleted. So we save on the partitions loaded by the cluster. This feature makes sure
+a topic can only be deleted when there are no subscriptions for the topic. Hence, no Producers and Consumers are affected by deleting.
+This deletion request can be triggered only by Topic owner teams. Please note, it follows the same request/approval process.
+
+Delete Acls
+-----------
+
+It is possible to delete particular Producer or Consumer subscriptions of a topic with this feature. This deletion request can be
+triggered by that Acl owner teams. No other teams can request for other's acl deletion. So all the subscriptions are secure this way.
+
+
+Ldap Authentication/Active Directory
+------------------------------------
+
+Users can be authenticated against Ldap/Active Directory while users login to Kafkawize.
+
+If you do not want to create/use a different password for Kafkawize users, rather authenticate from an existing Ldap server of your
+organization, it is easy to integrate now. Application properties have properties like baseDN, userDNPattern, passwordAttributes etc
+to configure your Ldap server and connect.  The below property enables ldap authentication::
+
+Cluster Connect Protocols
+-------------------------
+
+kafkawizeclusterapi can connect to Kafka cluster over the below protocols.
+
+PLAINTEXT
+~~~~~~~~~
+
+Kafkawizeclusterapi connects can connect to Kafka clusters over PLAINTEXT protocol
+When a cluster is being created in Kafkawize, this protocol can be selected.
+
+SSL
+~~~
+
+Kafkawizeclusterapi connects can connect to Kafka clusters over SSL protocol
+When a cluster is being created in Kafkawize, this protocol can be selected. Make sure SSL parameters are configured in application.properties
+in Kafkawizeclusterapi.
+SSL Parameters like keystore, truststore, passwords, etc.
+
+SASL_PLAIN
+~~~~~~~~~~
+
+Kafkawizeclusterapi connects can connect to Kafka clusters over SASL_PlAIN protocol
+When a cluster is being created in Kafkawize, this protocol can be selected. Make sure SSL parameters are configured in application.properties
+in Kafkawizeclusterapi.
+
+SASL stands for Simple Authentication and Security Layer. Kafka Jaas configuration has to be configured on the Kafka cluster
+with username/passwords.
+Kafkawizeclusterapi application.properties would be configured with SASL Jaas config too like below::
+
+    org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='admin-secret';
+
+
+SASL_SSL
+~~~~~~~~
+
+Kafkawizeclusterapi connects can connect to Kafka clusters over SASL_SSL protocol
+When a cluster is being created in Kafkawize, this protocol can be selected. Make sure SSL parameters are configured in application.properties
+in Kafkawizeclusterapi.
+
+SASL stands for Simple Authentication and Security Layer. Kafka Jaas configuration has to be configured on the Kafka cluster
+with username/passwords.
+Kafkawizeclusterapi application.properties would be configured with SASL Jaas config too like below
+
+It is possible to configure PlainLoginModule or KerberosLoginModule::
+
+    com.sun.security.auth.module.Krb5LoginModule required \
+            useKeyTab=true \
+            storeKey=true \
+            keyTab="/etc/security/keytabs/kafka_client.keytab" \
+            principal="kafkaclient1@EXAMPLE.COM";
+
+SSL parameters should also be configured, when ssl encryption is enabled.
+
+SASL mechanism can be PLAIN or GSSAPI(Kerberos)
+
+
+Synchronize Metadata
+--------------------
+
+A situation where Kafka cluster already exists and would like to adapt Kafkawize in your organization, all the topics and acls need to have their teams/owners.
+
+This feature is possible with Synchronization of Topics or Acls.
+
+Topics
+~~~~~~
+After a environment is selected, topics are displayed, and a team can be assigned to it. And this action, team becomes the Owner team.
+
+.. image:: _static/images/SynchronizeTopics.JPG
+    :width: 500px
+    :align: center
+
+It is required to synchronize the topic team first with Base sync cluster first. Base sync cluster can be configured with property custom.syncdata.cluster in application properties.
+
+Acls
+~~~~
+After a environment is selected, Producer and Consumer Acls are displayed, and a team can be assigned to it. After this action, team becomes the Owner of that subscription (producer or consumer).
+
+.. image:: _static/images/SynchronizeAcls.JPG
+    :width: 500px
+    :align: center
+
+Restore Config
+--------------
+
+Configuration of topics and acls from metastore can be restored back on any selected Kafka cluster.
+
+Multi Tenancy
+-------------
+
+Each tenant can manage their topics with their own teams in isolation.
+Every tenant can have their own set of Kafka environments, and users of one tenant cannot view/access topics, acls
+or any from other tenants.
+It provides an isolation avoiding any security breach.
+
+Analytics
+---------
+
+Several charts are introduced to give an overview of Clusters and usages by teams.
+Topics, Acls and Partitions per team
+Topics, Acls and Partitions per environments
+Activity log per team and per environments
+
+Configurable Roles & Permissions
+--------------------------------
+
+Any new roles can be added and associate different kind of permissions
+Permissions
+A whole bunch of permissions can be assigned to roles from User interface, making it very flexible.
+With immediate effect, users will be able to see the changes.
+Ex: A permission to request topics, or approve or add users, environments, clusters etc.
+
+
+User Registration
+-----------------
+
+New users can register from the home page, and request is forwarded to Super user. After the approval, user is added to the system.
+
+
+Email Notifications
+-------------------
+For every request and approval, through out the Kafkawize application, emails are sent out to approvers, and after approvals,
+notify the requesters. When a new user is added, or password changes, etc, notifications are enabled.
+
+
+Kafka connectors
+----------------
+
+Create and approve Kafka connectors