RAPyDo 0.7.4 (#33)

* Bump version 0.7.4

* Deprecated getJsonResponse

* Dropped support for wrapped responses

* Dropped deprecated meta parameter in response

* Dropped deprecated use of commons in swagger specs

* Deprecated use of custom in swagger specs

* Dropped deprecated publish flag in swagger specs templates)

* Dropped deprecated html templates (already replaced with jinja2 

* Dropped deprecated get_logger function

* Dropped support for RecoverSchema

* Deprecated use of METHODs endpoints confs, replace with _METHODs

* Deprecate use of self.get_current_user, to be replaced with self.auth.get_user

* Dropped getJsonResponse and custom neomodel types

* Removed download.send_file_partial method, use normal download methods by sending Range header

* Moved BaseAuthentication initialization from Connector to Service

* Removed double db connection in mondo connector

* Replaced string concatenation and string formatting with f-strings

* Added decorator for init_chunk_upload

* Added get_backend_url and get_frontend_url utilities as replacement of previous (request based) get_api_url

* Let Authentication service to retrieve own variables from conf

* Merged check_configuration and load_classes in detector -> load_services

* Moved HandleSecurity from auth connector to auth service

* Removed Authentication connector

* Removed abackend parameter from connectors' initialization method

* Refactor of db initialization and destruction

* Removed prefix field from connectors configuration

* Normalized connectors name and env variables

* Removed worker_mode from detector init_services

* Moved Authentication classes from services to connectors

* Moved bearer from services/authentication to rest module

* Renamed services/authentication/__init__.py as services/authentication.py

* Added tasks injection from detector to celery connector

* Merged HandleSecurity and BaseAuthenticationService

* Bug fix in case of errors during smtp client initialization

* Bug fix SMTP_SSL: smtp port is now expected to be an int

* Renamed endpoints METHODs configuration into _METHOD

* Extended catch errors decorator to intercept AccessRefused errors raise by RabbitMQ

* Replaced deprecated use of self.get_current_user() with self.auth.get_user()

* Upgraded sentry-sdk from 0.14.3 to 0.14.4

* Upgraded flower from 0.9.3 to 0.9.4

* Upgraded celery-redbeat from 0.13.0 to 1.0.0

* Upgraded flask-apispec from 0.8.7 to 0.8.8

* Upgraded Werkzeug from 0.16.1 to 1.0.1

* Upgraded schemathesis from 1.2.0 to 1.6.2

* Bug fix: with new werkzeug all responses to Range requests have status 206, even if completed

* Removed attrs dependency

* Added is_active field as optional input in admin users endpoint (Default to True)

* Removed uWSGI dependency, now part of the docker image

* Removed temporary dump of swagger specs on file (previously used as a validation step?)

* Removed Werkzeug dependency

* Added Back-compatibility fix to make tests work with Werkzeug 0.16.1 (still used in backendirods)

* Removed unused custom_init hook

* Removed unused irods_user method from postgres connector

* Replaced os.environ.get with os.getenv

* Removed deprecated use of errors in response (to be replaced with raise RestApiException or response(content, code>=400))

* Moved models from models.service to connectors.service.models

* Set default role when creating / updating users. (used as default option when no role is provided)

* Fix to retrieve remote ip from gunicorn

* Split BaseAuthentication.myinit into load_default_user and load_roles

* Removed optional tests/ prefix from --file parameter used in restapi tests, this way the command will accept a full path as input

* Input schema in admin users endpoint is now dinamycally instanced at runtime

* Removed autocomplete support for groups [no longer used by the frontend]

* Replace brackets initialization with setdefault

* Raised endpoint errors during specs registrations

* Converted all core endpoints to flask-apispec

* Removed min pwd len from login input

* Removed swagger model from core

* Moved update properties utility from endpoint definition to db connectors

* Definitions is no longer a mandatory field in swagger spec

* Renamed rabbit connector as rabbitmq, to match name and env prefix

* Removed static connector configuration

* Added group coordinator to output data

* Added ISO8601UTC format used to integrate marshmallow schemas with javascript date

* Added Neo4jSchema for automatic conversion from Neo4j model to marshmallow schema. Example of use:
    class MySchema(Schema):
      x = fields.Nested(Neo4jSchema(mymodels.Test, fields=('a', 'b',)))

* Refactor of bearer implementation, now provided as utility class HTTPTokenAuth

* Introduced underscores in long number to improve readability (PEP 515)

* Removed deprecated force_response method

* Removed unused serialize and obj_serialize utilities (introduced with previous version as a temporary replacement for getJsonResponse)

* Dropped obsolete methods get_endpoint_custom_definition and read_properties

* Deprecated use of custom_parameters in swagger spec

* Removed unused utility string_from_timestamp

* Changed use of Downloader and Uploader to access static methods

* Removed use of hcodes

* Bug fix for smtp confs without port

* Fix to handle with long nested dict in logs

* Added Faker to tests [random data generator]

* Test buildData is now faker-based

* Injected Faker fixture with additional Password provider

* Added random locale to Faker initialization

* Defaulting is_active and email_notification to True when testing user creation

* Set utf8 extended collation for mysql

* Fix to prevent collation with postgres

* Spreading faker value all over the tests

* Extended mysql collation to surname field

* Bug fix to prevent duplicated tags in swagger spec

* Added utility to start and stop timeout signals

* Removed obsolete _parameter_schemas from swagger

* Deprecated use of get_paging method, replaced with new @decorators.get_pagination

* Dropped keep_order from load_yaml_file utility (now enabled by default)

* Removed unsupported output formats (xml, csv)

* Replaced default FlaskApiSpec swagger endpoint with custom implementation with ability to handle authentication and exclude private endpoint from public version

* Extended verify_token method to retrieve token validity, token, jti and user

* Replaced get token used in logout endpoint with a thread safe implementation

* Refactor of verify_token method to prevent private info stored in auth object

* Using jti from verify_token instead of auth._jti

* Added alternative get_user in definition to retrieve information from unpacked token instead of auth instance

* Removed access to _user from outside authentication package

* Filtering output private endpoints AND schemas in case of un-authenticated requests

* Fixed schemathesis to be able to fetch full swagger schema by providing valid credentials (replaced schemathesis.from_wsgi with schemathesis.from_dict)

* url-encoded token used to provide schemas to schemathesis

* Replaced custom authentication pre-schemathesis with standard login utility

* Fixed login used by schemathesis to be post-actions-aware [i.e. auto change password for expired or temporary passwords]

* Bug fix for correct duplication key detection in mongodb

* BUG fix: response from websocket stream is not a json but a plain text

* Replaced used of random with secrets (new in py36)

* Converted string formatting to f-string, fixed super() syntax, removed coding: utf-8

* Introduced RestApiException specializations based on specific status_code (e.g. Forbidden, BadRequest, etc)

* BUG fix: ip extraction from X-Forwarded-For was overwritten by ip obtained from request.remote_addr

* Pagination endpoint automatically change schema location based on decorated method

* Fix to prevent post to accept pagination parameters as query strings

* Added conf variable to limit log lenght (200 by default, decreased during tests)

* Added an Env module with utilities for env variable conversions. Deprecated analogous utilities in detector

* Utility to (re)create the logger, used before schemathesis to reduce the amount of generated logs by raising the log level to WARNING

* Specialized Schemas as Input and Output Schema to prevent get_schema special attributed to be inserted as output

* Removed get_schema parameters from definitions included in output specs

* Fix to include tags descriptions in new swagger spec

* Marshmallow schema response: missing attribute is now used as default field (fallback to default attribute as secondary choice)

* Token last access is now saved every 60 seconds to avoid excessive and unnecessary db resources wasting

* Fix to force image build when PR or cron

* Applied --destroy flag to tests. Moved destroy test as last test, after schemathesis

* Added checks to verify result of destroy and init modes

* Removed unused parameters from Meta.get_class_from_string utility

* Fixes to let SQL destroy mode during tests

* Bug fix in case of malformed content range during chunked upload

* Created parse_content_range utility to implement specific tests on expected behaviour

* Bug fix to correctly read REGISTER_FAILED_LOGIN conf

* Applied get_backend_url to init chunked upload

* Enabled tests in production mode (added variables to force tests in production). [very dangerous, only use it on volatile instances]

* Prevent destroy test in production

* Removed partially used retry method in connectors

* Disabled filter_too_much check in hypothesis

* Refactor of instances management in base connector

* Fixed alchemy connector to enabled servce availability

* Better management of connector variables to test failures

* Refactor of rabbit connector, removed lazy connections

* Added socket.gaierror to rabbit connection exceptions

* Added a parser for InternalErrors raised by sqlalchemy

* Refactor of irods connector: merged pre-connect and connect methods to simplify the connection workflow

* Added GSSErrors to irods connection exceptions

* Added FileNotFoundError to irods connection exceptions

* Replaced get_input with use_kwargs in endpoint implementation

* Added pre-commit configuration

* Applied isort, pyupgrade and black on all files

* Reduced McCabe complexity to 36

* Celery tests are now based on new rapydo add task command

* Refactor of connectors instance handling [now instanced that thread level]

* Refactor of connector connection close: moved from request to change expiration

* Extended cache expiration tests to all connectors

* Merged RabbitExt and RabbitWrapper

* Merged PushpinExt and PushpinClient

* Merged neomodel connector and neomodel client

* Merged IrodsPythonExt and IrodsPythonClient

* Extended support for with statement to all connectors

* Moved alchemy instance from models to connector

* Fixed alchemy models to correctly match tokens between different threads

* Celery tasks injection is now executed at instance level instead of class level

* Automatic connector re-initialization in case of forced disconnection

* Fix celery connector to return standard response

* Added utilities to check, create and delete rabbit queues

* Reimplemented a get_debug_instance utility to detector

* Set explicit parameters in connector get_instance

* Extended user profile output schema with custom fields

* Removed unused islink parameter in update_properties utilities

* Refactor of initializer and customizer instances loading

* Fixed checks for inactive credentials on mysql

* Added TOTP libs

* Re-enabled 2FA TOTP

* Fix to allow password reset with totp

* Bug fix to prevent password change when TOTP is required but missing

* Fix to make schemathesis work with totp authentication

* Extended TOTP validity to +/1 current time

* Bug fix to prevent missing totp to hide first login actions

* Suppressed warning from apispec about Multiple schemas

* Added prettier to pre-commit conf

* Applied prettier formatter to all files

* Added a UniqueDelimitedList by extending webargs DelimitedList with custom validation to reject lists with duplicated values

* Fixed deprecated use of verify_admin in auth

* Mock mails are now saved onto the logs folder

* Moved celery logs into a subfolder in data/logs

* Bug fix in init Authentication roles and users

* Removed OrderedDict, from python 3.7 normal dict are guarantee to preserve keys order

* Dropped unused exception parameter from catch_errors

* Bug fixed: user registration was not considering the password confirmation

* Bug fixed: user registration was not considering the password strength