Error during refresh metadata

[sid88in]

2016-04-28 17:57:16,788:INFO:werkzeug:66.228.162.56 - - [28/Apr/2016 17:57:16] "GET /caravel/refresh_datasources/ HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/y/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/y/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/home/y/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/y/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/y/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/y/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/y/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/y/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/y/lib/python2.7/site-packages/flask_appbuilder/security/decorators.py", line 26, in wraps
    return f(self, *args, **kwargs)
  File "/home/y/lib/python2.7/site-packages/caravel/views.py", line 816, in refresh_datasources
    cluster, str(e)),
  File "/home/y/lib/python2.7/site-packages/caravel/models.py", line 825, in __repr__
    return self.cluster_name
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 578, in get
    value = state._load_expired(state, passive)
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line 474, in _load_expired
    self.manager.deferred_scalar_loader(self, toload)
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 664, in load_scalar_attributes
    only_load_props=attribute_names)
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident
    return q.one()
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2693, in one
    ret = list(self)
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
    return self._execute_and_instances(context)
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
    close_with_result=True)
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
    **kw)
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 893, in connection
    execution_options=execution_options)
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
    engine, execution_options)
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 313, in _connection_for_bind
    self._assert_active()
  File "/home/y/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 214, in _assert_active
    % self._rollback_exception
InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (_mysql_exceptions.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`caravel`.`columns`, CONSTRAINT `columns_ibfk_1` FOREIGN KEY (`column_name`) REFERENCES `datasources` (`datasource_name`))') [SQL: u'INSERT INTO `columns` (created_on, changed_on, datasource_name, column_name, is_active, type, groupby, count_distinct, sum, max, min, filterable, description, changed_by_fk, created_by_fk) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (datetime.datetime(2016, 4, 28, 17, 57, 16, 765009), datetime.datetime(2016, 4, 28, 17, 57, 16, 765018), 'fact_search_term', 'conversions', 1, 'LONG', 0, 0, 0, 0, 0, 0, None, 1L, 1L)]

[spektom]

I'm getting the same error:

2016-05-03 08:52:27,858:DEBUG:requests.packages.urllib3.connectionpool:"GET /druid/coordinator/v1/metadata/datasources HTTP/1.1" 200 None
Syncing Druid datasource [activities]
Traceback (most recent call last):
  File "/usr/local/bin/caravel", line 109, in <module>
    manager.run()
  File "/usr/local/lib/python2.7/dist-packages/flask_script/__init__.py", line 412, in run
    result = self.handle(sys.argv[0], sys.argv[1:])
  File "/usr/local/lib/python2.7/dist-packages/flask_script/__init__.py", line 383, in handle
    res = handle(*args, **config)
  File "/usr/local/lib/python2.7/dist-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "/usr/local/bin/caravel", line 99, in refresh_druid
    cluster, str(e)))
  File "/usr/local/lib/python2.7/dist-packages/caravel/models.py", line 863, in __repr__
    return self.cluster_name   
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 578, in get
    value = state._load_expired(state, passive)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 474, in _load_expired
    self.manager.deferred_scalar_loader(self, toload)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 664, in load_scalar_attributes
    only_load_props=attribute_names)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident
    return q.one()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2693, in one
    ret = list(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
    close_with_result=True)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
    **kw)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
    execution_options=execution_options)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
    engine, execution_options) 
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 313, in _connection_for_bind
    self._assert_active()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 214, in _assert_active
    % self._rollback_exception 
sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.IntegrityError) insert or update on table "columns" violates foreign key constraint "columns_column_name_fkey"
DETAIL:  Key (column_name)=(campaign) is not present in table "datasources".
 [SQL: 'INSERT INTO columns (created_on, changed_on, datasource_name, column_name, is_active, type, groupby, count_distinct, sum, max, min, filterable, description, changed_by_fk, created_by_fk) VALUES (%(created_on)s, %(changed_on)s, %(datasource_name)s, %(column_name)s, %(is_active)s, %(type)s, %(groupby)s, %(count_distinct)s, %(sum)s, %(max)s, %(min)s, %(filterable)s, %(description)s, %(changed_by_fk)s, %(created_by_fk)s) RETURNING columns.id'] [parameters: {'description': None, 'min': False, 'filterable': True, 'max': False, 'sum': False, 'is_active': True, 'created_on': datetime.datetime(2016, 5, 3, 8, 52, 27, 898158), 'count_distinct': False, 'changed_by_fk': None, 'datasource_name': u'activities', 'created_by_fk': None, 'type': u'STRING', 'groupby': True, 'changed_on': datetime.datetime(2016, 5, 3, 8, 52, 27, 898172), 'column_name': u'campaign'}]

We're using PostgreSQL as metadata database.
This is the workaround that worked for us:

alter table columns drop constraint columns_column_name_fkey;

[mistercrunch]

Maybe it's a matter of adding a session.flush() after creating the DruidDatasource. Can you try this? #430 . If it fixes the problem I'll merge it.