Updates on Installation

index.rst

@@ -18,5 +18,5 @@ Contents:
     Big Data Platform <bigdataplatform/index>
     Data Management <datamgmt/index>
     Microservices <microservices/index>
-    Docker recipies <docker/index>
+    Installation  <installation/index>
     [ITA] Manuale utente <manutente/index>

installation/docker/ckan.rst

@@ -0,0 +1,123 @@
+
+Ckan
+============================================================
+
+CKAN is an open-source DMS (data management system) for powering data hubs and data portals.
+CKAN makes it easy to publish, share and use data. It powers datahub.io, catalog.data.gov and data.gov.uk among many other sites.
+
+This guide will show you how to use Docker Compose to set up and run a `Ckan <https://ckan.org/>`_
+ instance which uses ldap credentials to authenticate users. In particular, you can use an openLDAP docker container or a freeIpa instance.
+
+OpenLDAP docker compose
+-----------------------
+
+This docker container allows us to start a simple LDAP server (`OpenLdap <http://www.openldap.org/>`_
+) and a client (`phpLDAPadmin <http://phpldapadmin.sourceforge.net/>`_
+). In particular, the docker compose downloads an initial database having domain *daf.test.it* and containing the user *bob* with password *password*.
+
+Clone the git project:
+
+.. code-block:: bash
+
+ > git clone git@github.com:italia/daf-recipes.git
+
+Run the docker container:
+
+.. code-block:: bash
+
+  > cd ./daf-recipes/ldap
+  > docker-compose up -d
+
+Check whether dockers are running:
+
+.. code-block:: bash
+
+  > docker ps
+  e8ff9611aeff        osixia/openldap         "/container/tool/r..."   17 minutes ago      Up 17 minutes       0.0.0.0:389->389/tcp, 0.0.0.0:636->636/tcp   ldap
+  6a0d0d6c3b9a        osixia/phpldapadmin     "/container/tool/run"    17 minutes ago      Up 17 minutes       0.0.0.0:80->80/tcp, 443/tcp                  phpldapadmin
+
+**Note**
+
+The docker compose requires that ports 80, 636 and 389 are available. If not, change them.
+
+
+Now, open your favorite browser and type *http://localohost*. Login as *cn=admin,dc=example,dc=org* and password *admin* to navigate inside.
+
+.. image:: imgs/ldap_login.png
+   :scale: 50 %
+   :alt: alternate text
+   :align: right
+
+.. image:: imgs/ldap_tree.png
+   :scale: 50 %
+   :alt: alternate text
+   :align: right
+
+FreeIpa Instance
+-----------------------
+We installed a FreeIpa server which can be used for test purposes. It can be reached to the address *91.206.129.245*
+
+
+Ckan docker compose
+-----------------------
+
+Now that we have a ldap server up we can run the Ckan docker compose. It will run an instance of Solr, Postgresql, Redis and Mongo.
+
+First we have to build some custom image:
+
+
+.. code-block:: bash
+
+  > cd ./daf-recipes/ckan
+  > ./build_local.sh
+
+Then edit the file *ckan.ini*:
+
+- If you are using our openLDAP server:
+
+   .. code-block:: bash
+
+      # LDAP Intergration with ldap and ip address
+      ckanext.ldap.uri = ldap://LDAP_IP:389
+      ckanext.ldap.auth.dn = cn=admin,dc=daf,dc=test,dc=it
+      ckanext.ldap.auth.password = admin
+      ckanext.ldap.base_dn = cn=users,cn=accounts,dc=daf,dc=test,dc=it
+      ckanext.ldap.search.filter = uid={login}
+      ckanext.ldap.username = uid
+      ckanext.ldap.email = mail
+      ckanext.ldap.ckan_fallback = True
+
+where LDAP_IP is the ip of the LDAP docker. To know the LDAP ip run:
+
+.. code-block:: bash
+
+  > docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ldap
+  172.22.0.2
+
+We know that this is not the best approach to connect container among them (maybe it is the worst), we are using a deprecated compose file version (i.e. version 1 rather than using version 3), and we are using very heavy images. We will improve Ckan docker compose as soon as possible.
+
+- If you are using our FreeIpa server
+     .. code-block:: bash
+
+      # LDAP Intergration with ldap and ip address
+      ckanext.ldap.uri = ldap://91.206.129.245:389
+      ckanext.ldap.auth.dn = uid=admin,cn=users,cn=accounts,dc=daf,dc=test,dc=it
+      ckanext.ldap.auth.password = aiyaiPh8
+      ckanext.ldap.base_dn = cn=users,cn=accounts,dc=daf,dc=test,dc=it
+      ckanext.ldap.search.filter = uid={login}
+      ckanext.ldap.username = uid
+      ckanext.ldap.email = mail
+      ckanext.ldap.ckan_fallback = True
+
+
+Now that ckan container is up, type *http://localhost:5000* on your browser and login as user *bob* (password *password*).
+
+.. image:: imgs/ckan_login.png
+   :scale: 50 %
+   :alt: alternate text
+   :align: right
+
+.. image:: imgs/bob_page.png
+   :scale: 50 %
+   :alt: alternate text
+   :align: right

installation/docker/freeipa.rst

@@ -0,0 +1,79 @@
+
+Freeipa LDAP version: 4.4.0
+=========================================
+
+Docker
+-----------------
+FreeIPA server can be run in a Docker container for testing or demo purposes. It makes it possible to run all the processes comprising the server in an isolated way, leaving the host free to run other software, not clashing with the FreeIPA server.
+
+This install is done on Ubuntu 16.04. FreeIPA is focused on Linux (and other standards compliant) systems. FreeIPA is focused on Linux (and other standards compliant) systems. Therefore, in our knowledge, you cannot run a container of a FreeIPA server on **Mac** or **Windows**. However, any help in this direction is very welcomed!!
+
+
+Follow these steps to run our FreeIPA server docker:
+
+1 Create a directory which will hold the server data:
+
+.. code-block:: bash
+
+   mkdir /var/lib/ipa-data
+
+2  Edit */etc/hosts* and ensure that the IPA server address is listed. This is required for Apache to work properly. You have to change IPA_SERVER_IP with the ipa server ip:
+
+.. code-block:: bash
+
+       127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
+       ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
+       IPA_SERVER_IP	ipa.example.test
+
+
+3 Finally, You run the container:
+
+.. code-block:: bash
+
+    docker run -it -p 389:389 -p 443:443 -p 636:636 --name freeipaldap --cap-add SYS_ADMIN --security-opt seccomp:unconfined -v /sys/fs/cgroup:/sys/fs/cgroup:ro --tmpfs /run --tmpfs /tmp -v /var/lib/ipa-data:/data:Z -h ipa.example.test italia/freeipa-server --ds-password=The-directory-server-password --admin-password=The-admin-password
+
+where:
+
+- *--cap-add SYS_ADMIN*, performs a range of system administration operations (see `here <https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities>`_ for more details ).
+- *--security-opt seccomp:unconfined* to run a container without the default seccomp profile (see `here <https://docs.docker.com/engine/security/seccomp/>`_ for more details ).
+- *-v /var/lib/ipa-data:/data:Z* to store data and configurations in the folder */var/lib/ipa-data/*
+Answer to the question:
+
+        Do you want to configure integrated DNS (BIND)? [no]:   --> press "Enter"
+
+        Server host name [ipa.example.test]:                    --> press "Enter"
+
+        Please confirm the domain name [example.test]:          --> press "Enter"
+
+        Please provide a realm name [EXAMPLE.TEST]:             --> press "Enter"
+
+        Continue to configure the system with these values? [no]:  --> type "y" and press "Enter"
+
+Wait some time until freeipa server is completely configured and started.
+The server is ready when on the shell appear the followuing message:
+
+        FreeIPA server configured.
+
+NOTE: Only first time that build image and run docker you need to ask to previous questions.
+
+- You can connect to Freeip Server with web interface:
+        https://IPA_SERVER_IP:443
+        USER admin
+        PW adminpassword
+
+- You can also connect with LDP client with Server IP address IPA_SERVER_IP
+
+- The container can the be started and stopped with the following commands:
+
+        docker stop freeipaldap
+
+        docker start freeipaldap
+
+
+References
+-----------------
+[1] `FreeIpa docker-hub documentation <https://hub.docker.com/r/freeipa/freeipa-server/>`_.
+
+[2] `Using Free Ipa for user authentication <https://annvix.com/using_freeipa_for_user_authentication>`_.
+
+[3] `FreeIpa website <https://www.freeipa.org/page/Docker>`_.

installation/docker/index.rst

@@ -0,0 +1,16 @@
+
+Docker recipies of open source tools used in DAF
+=================================================
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+    Install step-by-step <install>
+    Ckan       <ckan>
+    Freeipa    <freeipa>
+    Jupyter    <jupyter>
+    LDAP       <ldap>
+    Metabase   <metabase>
+    Superset   <superset>

installation/docker/install.rst

@@ -0,0 +1,30 @@
+
+Install
+============================================================
+
+This installation is tested on Ubuntu. The main reason is the difficulties to
+launch freeipa docker on mac and windows (Any help making official freeipa docker 
+working on these platforms will be very usefull). For users with mac and windows 
+that want to test the application faster I suggest to install freeipa in some cloud 
+provider with a public ip and ldap ports opened.
+
+For angry users it is possible to install directly this virtul box images (wo)
+
+These are a series of docker recipes that are used in daf-dataportal for giving tools and 
+instruments for analyzing and visualizing data. 
+
+In not necessary to install all dockers in order but is important to install freeipa as 
+first docker because is an identity manager system which all dockers connect to have access.
+
+- Install freeipa following the instructions in freeipa section.
+
+- Install ckan following the instructions in ckan section.
+
+- Install superset following the instructions in superset section.
+
+- Install metabase following the instructions in metabase section.
+
+- Install jupyter (working progress)
+
+
+

installation/docker/jupyter.rst

@@ -0,0 +1,78 @@
+
+Jupyter
+============================================================
+
+This guide will show you how to use Docker Compose to set up and run a `JupyterHub <https://jupyterhub.readthedocs.io/en/latest/>`_  instance
+which uses ldap credentials to authenticate users.
+
+LDAP
+-----------------
+This docker container allows us to start a simple LDAP server (`OpenLdap <http://www.openldap.org/>`_
+) and a client (`phpLDAPadmin <http://phpldapadmin.sourceforge.net/>`_
+). In particular, the docker compose download an initial database, contained the user *alice* having password *password*.
+
+Clone the git project:
+
+.. code-block:: bash
+
+ > git clone git@github.com:italia/daf-recipes.git
+
+Run the docker container:
+
+.. code-block:: bash
+
+  > cd ./daf-recipes/ldap
+  > docker-compose up -d
+
+Check whether dockers are running:
+
+.. code-block:: bash
+
+  > docker ps
+  e8ff9611aeff        osixia/openldap         "/container/tool/r..."   17 minutes ago      Up 17 minutes       0.0.0.0:389->389/tcp, 0.0.0.0:636->636/tcp   ldap
+  6a0d0d6c3b9a        osixia/phpldapadmin     "/container/tool/run"    17 minutes ago      Up 17 minutes       0.0.0.0:80->80/tcp, 443/tcp                  phpldapadmin
+
+**Note**
+
+The docker compose requires that ports 80, 636 and 389 are available. If not, change them.
+
+
+Now, open your favorite browser and type *http://localohost*. Login as *cn=admin,dc=example,dc=org* and password *admin* to navigate inside.
+
+.. image:: imgs/ldap_login.png
+   :scale: 50 %
+   :alt: alternate text
+   :align: right
+
+.. image:: imgs/ldap_tree.png
+   :scale: 50 %
+   :alt: alternate text
+   :align: right
+
+
+JupyterHub
+-----------------
+
+This docker container runs a JupyterHub instance which is connected with a postgresql database.
+
+Run the docker container:
+
+.. code-block:: bash
+
+  > cd ./daf-recipes/jupyterhub
+  > docker-compose up -d
+
+Check whether dockers are running:
+
+.. code-block:: bash
+
+  > docker ps
+  8350963ac06c        jupyterhub_jupyterhub   "/wait_db_is_ready.sh"   16 minutes ago      Up 16 minutes       0.0.0.0:8000->8000/tcp                       jupyterhub
+  6a0d0d6c3b9a        osixia/phpldapadmin     "/container/tool/run"    17 minutes ago      Up 17 minutes       0.0.0.0:80->80/tcp, 443/tcp                  phpldapadmin
+  e8ff9611aeff        osixia/openldap         "/container/tool/r..."   17 minutes ago      Up 17 minutes       0.0.0.0:389->389/tcp, 0.0.0.0:636->636/tcp   ldap
+  cee2d35feaaf        postgres:9.6            "docker-entrypoint..."   2 hours ago         Up 2 hours          0.0.0.0:5432->5432/tcp                       postgresjupyterhub
+
+To open the interactive shell type *http://localhost:8000* and login as user *alice* (password *password*).
+
+.. image:: imgs/jupyter.png
+

installation/docker/ldap.rst

@@ -0,0 +1,5 @@
+*****************
+LDAP Installation
+*****************
+
+

installation/docker/metabase.rst

@@ -0,0 +1,17 @@
+
+Metabase
+============================================================
+
+Metabase + postgres + ldap configuration
+
+Follow these steps to run the Docker images:
+
+cd metabase
+
+./build_local.sh        #it will build the images needed by docker-compoose
+
+docker-compose up -d    # it will run all the needed containers
+
+Then you can open the metabase home http://localhost:3000
+
+goes to `github <https://github.com/italia/daf-recipes/tree/master/metabase>`_ for seeing how to setup metabase

installation/docker/superset.rst

@@ -0,0 +1,33 @@
+
+Superset
+============================================================
+
+cd superset
+
+Configure LDAP parameter inside the configuration file "superset_config.py" before to build superset image
+
+ - AUTH_TYPE = AUTH_LDAP
+ - AUTH_LDAP_SERVER = "ldaps://server:636"
+ - AUTH_LDAP_SEARCH = "cn=users,cn=accounts,dc=test,dc=example,dc=it"
+ - AUTH_LDAP_UID_FIELD = "uid"
+ - AUTH_LDAP_FIRSTNAME_FIELD = "givenName"
+ - AUTH_LDAP_LASTNAME_FIELD = "sn"
+ - AUTH_LDAP_EMAIL_FIELD = "mail"
+ - AUTH_LDAP_BIND_USER = "uid=admin,cn=users,cn=accounts,dc=test,dc=example,dc=it"
+ - AUTH_LDAP_BIND_PASSWORD = "password"
+ - AUTH_LDAP_ALLOW_SELF_SIGNED = True
+
+
+#Build superset image
+
+./build.sh 
+
+docker-compose up -d 
+
+It will start superset - postgres - redis
+
+wait some time to be sure all docker will be up and running
+
+./init.sh 
+
+#First time to create admin user (admin password) - initialize db - exmaple