Skip to content
first draft for solidata_backend API
Jupyter Notebook Python HTML
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.vscode
_notebooks
admin
logs
prod_snippets
screenshots
solidata_api
templates/emails
uploads/data_sources
.flaskenv
.gitignore
CHANGELOG.md
CONTRIBUTING.md
GUIDELINES_DEV.md
LICENCE
README.md
appserver.py
log_config.py
manage.py
requirements.txt
wsgi.py
wsgi_preprod.py

README.md

SOLIDATA
backend

WARNING : almost deprecated

From now please refer to the cloned and enhanced repo : solidata-backend


PRESENTATION

part 2a/3 of the TADATA! sofware suite (ApiViz / Solidata_frontend / OpenScraper )

Building a public service for managing, consolidating, and sharing "api-fied" data


GOALS

  • allow you to consume data from your own csv/xls files or external APIs ;
  • consolidate your data : apply your own datamodel, simplify columns, apply metadatas, ... ;
  • share the consolidated datas on API endpoints with the level of openness you decide (opendata, commons, collective, private) ;
  • manage all your data and your recipes by projects ;
  • be able to manage projects by teams and share data/recipes/datamodels...

DEVELOPPERS

Nice to see you around ! For a more complete tour check out :


TECHNICAL POINTS

Tech stack

  • Language : Python 3.6... yes ... I know ... Python again ... hmmm ... gnnn ... don't judge me ?!
  • Framework : Flask... minimalistic Python framework
  • API : Flask-RestPlus... Swagger documentation integrated
  • Security : Flask-JWT-extended... wrapper JWT for Flask
  • Emailing : Flask-email... templating, sending, etc...
  • Data management : Pandas... to do operations on datasets...
  • Geocoding : Geopy... to use several geocoders : Nominatim, BAN, etc...

Features :

  • JWT (JSON Web Tokens) :
    • access and refresh token for security over all the app
  • user :
    • anonymous login (automatic)
    • login / register user
    • confirm user by sending a confirmation link (protected) in an email
    • password forgotten by sending a link (protected) in an email with redirection to new password form
    • reset password from client interface (protected) ...
  • Documentation
    • on all API endpoints with Swagger (and some patience from the developer)
Current features TO DO :
  • user :
    • edit user
    • edit email
  • add/edit a project :
    • edit team and access levels
    • add project to users involved

WHAT'S NEW ?

  • check out our our changelog for more details

INSTALLATION WALKTHROUGH

LOCALLY

  • clone / fork the depo ...

  • create a virtual environment for Python3

     python3 -m venv venv
     pip install --upgrade pip
     pip install -r requirements.txt
  • create a secret config_file config_prod.py in the folder ./solidata_api based

     cp ./solidata_api/config_pro_exampled.py ./solidata_api/config_prod.py
     nano ./solidata_api/config_prod.py 
  • pay attention at the MONGO_URI variable depending on your local mongodb configuration...

    • The following is the standard URI connection scheme (from mongo documentation):
       mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]
      
  • run the app in dev mode (without possibility to send email) :

     python appserver.py --mode=dev
  • run the app in dev mode (with possibility to send email) :

     python appserver.py --mode=dev_email

PRODUCTION

  • Check our snippets folder for more instructions

  • for now we are using the following configuration

    • droplet in digitalocean.com
    • ubuntu 18.04
    • 3Go RAM / 2CPU
    • 60Go memory
  • configure your server (user, firewall...):

  • install mongodb :

  • install nginx :

  • install git on your server

    • cf : docs 1 - basically :
       sudo apt-get update
       sudo apt-get install git
       git --version
    • go to your directory and init git :
       git init . 
       git remote add origin https://github.com/entrepreneur-interet-general/solidata_backend.git
       git pull origin master
      
  • same steps than for local installation (virtual env, install dependencies, config_prod.py file, ) ...

  • run the app in production mode (with possibility to send email) :

     python appserver.py --mode=production

INSPIRATIONS / BENCHMARK

  • Dataiku, Parabola, but they are proprietary solutions (and too complex for our pruposes)... Not to mention they are very expensive...
  • Match ID, but the backend doesn't resolve the "sharing" part with levels as ["opendata", "commons", "collective", "private"], and no BDD to back up recipes

CREDITS

Solidata's backend team thanks :

Contacts - maintainance :


SCREENSHOTS


alt text


alt text


alt text


alt text

You can’t perform that action at this time.