Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Problem with database update: ImportError: cannot import name execute_batch #5085

Closed
silvio opened this issue Apr 21, 2019 · 3 comments
Closed

Problem with database update: ImportError: cannot import name execute_batch #5085

silvio opened this issue Apr 21, 2019 · 3 comments

Comments

@silvio
Copy link

silvio commented Apr 21, 2019

Description

I have a problem with latest update of matrix:

Apr 21 11:13:42 sfserver docker[15688]: 2019-04-21 09:13:42,543 - synapse.storage.background_updates - 113 - ERROR - background_updates-0- Error doing update                                                                                                                                                                 
Apr 21 11:13:42 sfserver docker[15688]: Traceback (most recent call last):
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/background_updates.py", line 110, in _run_background_updates                                                                                                                                                           
Apr 21 11:13:42 sfserver docker[15688]:     self.BACKGROUND_UPDATE_DURATION_MS
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Apr 21 11:13:42 sfserver docker[15688]:     result = result.throwExceptionIntoGenerator(g)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Apr 21 11:13:42 sfserver docker[15688]:     return g.throw(self.type, self.value, self.tb)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/background_updates.py", line 185, in do_next_background_update                                                                                                                                                         
Apr 21 11:13:42 sfserver docker[15688]:     res = yield self._do_background_update(update_name, desired_duration_ms)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Apr 21 11:13:42 sfserver docker[15688]:     result = result.throwExceptionIntoGenerator(g)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Apr 21 11:13:42 sfserver docker[15688]:     return g.throw(self.type, self.value, self.tb)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/background_updates.py", line 219, in _do_background_update                                                                                                                                                             
Apr 21 11:13:42 sfserver docker[15688]:     items_updated = yield update_handler(progress, batch_size)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Apr 21 11:13:42 sfserver docker[15688]:     result = result.throwExceptionIntoGenerator(g)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Apr 21 11:13:42 sfserver docker[15688]:     return g.throw(self.type, self.value, self.tb)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/user_directory.py", line 216, in _populate_user_directory_process_rooms                                                                                                                                                
Apr 21 11:13:42 sfserver docker[15688]:     yield self.add_users_in_public_rooms(room_id, to_insert)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Apr 21 11:13:42 sfserver docker[15688]:     result = result.throwExceptionIntoGenerator(g)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Apr 21 11:13:42 sfserver docker[15688]:     return g.throw(self.type, self.value, self.tb)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 408, in runInteraction
Apr 21 11:13:42 sfserver docker[15688]:     *args, **kwargs
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Apr 21 11:13:42 sfserver docker[15688]:     result = result.throwExceptionIntoGenerator(g)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Apr 21 11:13:42 sfserver docker[15688]:     return g.throw(self.type, self.value, self.tb)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 457, in runWithConnection
Apr 21 11:13:42 sfserver docker[15688]:     inner_func, *args, **kwargs
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 250, in inContext
Apr 21 11:13:42 sfserver docker[15688]:     result = inContext.theWork()
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
Apr 21 11:13:42 sfserver docker[15688]:     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
Apr 21 11:13:42 sfserver docker[15688]:     return self.currentContext().callWithContext(ctx, func, *args, **kw)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
Apr 21 11:13:42 sfserver docker[15688]:     return func(*args,**kw)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py", line 306, in _runWithConnection
Apr 21 11:13:42 sfserver docker[15688]:     compat.reraise(excValue, excTraceback)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py", line 297, in _runWithConnection
Apr 21 11:13:42 sfserver docker[15688]:     result = func(conn, *args, **kw)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 453, in inner_func
Apr 21 11:13:42 sfserver docker[15688]:     return func(conn, *args, **kwargs)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 329, in _new_transaction
Apr 21 11:13:42 sfserver docker[15688]:     r = func(txn, *args, **kwargs)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/user_directory.py", line 547, in _add_users_in_public_rooms_txn
Apr 21 11:13:42 sfserver docker[15688]:     value_values=None,
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 813, in _simple_upsert_many_txn
Apr 21 11:13:42 sfserver docker[15688]:     txn, table, key_names, key_values, value_names, value_values
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 890, in _simple_upsert_many_txn_native_upsert
Apr 21 11:13:42 sfserver docker[15688]:     return txn.execute_batch(sql, args)
Apr 21 11:13:42 sfserver docker[15688]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 112, in execute_batch
Apr 21 11:13:42 sfserver docker[15688]:     from psycopg2.extras import execute_batch
Apr 21 11:13:42 sfserver docker[15688]: ImportError: cannot import name execute_batch

Version information

  • Homeserver:

If not matrix.org:

  • Version:
% curl -v https://matrix.port1024.net/_matrix/client/versions                                                                                                                                                                                              
*   Trying 2a03:b0c0:3:d0::3e:4001...                                                                                                                                                                                                                                                                                          
* TCP_NODELAY set                                                                                                                                                                                                                                                                                                              
* connect to 2a03:b0c0:3:d0::3e:4001 port 443 failed: Connection refused                                                                                                                                                                                                                                                       
*   Trying 159.89.26.70...                                                                                                                                                                                                                                                                                                     
* TCP_NODELAY set                                                                                                                                                                                                                                                                                                              
* Connected to matrix.port1024.net (159.89.26.70) port 443 (#0)                                                                                                                                                                                                                                                                
* ALPN, offering h2                                                                                                                                                                                                                                                                                                            
* ALPN, offering http/1.1                                                                                                                                                                                                                                                                                                      
* successfully set certificate verify locations:                                                                                                                                                                                                                                                                               
*   CAfile: /etc/ssl/certs/ca-certificates.crt                                                                                                                                                                                                                                                                                 
  CApath: none                                                                                                                                                                                                                                                                                                                 
* TLSv1.3 (OUT), TLS handshake, Client hello (1):                                                                                                                                                                                                                                                                              
* TLSv1.3 (IN), TLS handshake, Server hello (2):                                                                                                                                                                                                                                                                               
* TLSv1.2 (IN), TLS handshake, Certificate (11):                                                                                                                                                                                                                                                                               
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):                                                                                                                                                                                                                                                                       
* TLSv1.2 (IN), TLS handshake, Server finished (14):                                                                                                                                                                                                                                                                           
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):                                                                                                                                                                                                                                                                      
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):                                                                                                                                                                                                                                                                    
* TLSv1.2 (OUT), TLS handshake, Finished (20):                                                                                                                                                                                                                                                                                 
* TLSv1.2 (IN), TLS handshake, Finished (20):                                                                                                                                                                                                                                                                                  
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384                                                                                                                                                                                                                                                                   
* ALPN, server accepted to use h2                                                                                                                                                                                                                                                                                              
* Server certificate:                                                                                                                                                                                                                                                                                                          
*  subject: CN=port1024.net                                                                                                                                                                                                                                                                                                    
*  start date: Apr  1 03:00:41 2019 GMT                                                                                                                                                                                                                                                                                        
*  expire date: Jun 30 03:00:41 2019 GMT                                                                                                                                                                                                                                                                                       
*  subjectAltName: host "matrix.port1024.net" matched cert's "matrix.port1024.net"                                                                                                                                                                                                                                             
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3                                                                                                                                                                                                                                                                
*  SSL certificate verify ok.                                                                                                                                                                                                                                                                                                  
* Using HTTP2, server supports multi-use                                                                                                                                                                                                                                                                                       
* Connection state changed (HTTP/2 confirmed)                                                                                                                                                                                                                                                                                  
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0                                                                                                                                                                                                                                               
* Using Stream ID: 1 (easy handle 0x55648227ef30)                                                                                                                                                                                                                                                                              
> GET /_matrix/client/versions HTTP/2                                                                                                                                                                                                                                                                                          
> Host: matrix.port1024.net                                                                                                                                                                                                                                                                                                    
> User-Agent: curl/7.64.1                                                                                                                                                                                                                                                                                                      
> Accept: */*                                                                                                                                                                                                                                                                                                                  
>                                                                                                                                                                                                                                                                                                                              
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!                                                                                                                                                                                                                                                                    
< HTTP/2 200                                                                                                                                                                                                                                                                                                                   
< server: nginx/1.14.2                                                                                                                                                                                                                                                                                                         
< date: Sun, 21 Apr 2019 09:36:13 GMT                                                                                                                                                                                                                                                                                          
< content-type: application/json                                                                                                                                                                                                                                                                                               
< content-length: 187                                                                                                                                                                                                                                                                                                          
< access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS                                                                                                                                                                                                                                                                
< cache-control: no-cache, no-store, must-revalidate                                                                                                                                                                                                                                                                           
< access-control-allow-origin: *                                                                                                                                                                                                                                                                                               
< access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept, Authorization                                                                                                                                                                                                                                  
<                                                                                                                                                                                                                                                                                                                              
{                                                                                                                                                                                                                                                                                                                              
    "unstable_features": {                                                                                                                                                                                                                                                                                                     
        "m.lazy_load_members": true                                                                                                                                                                                                                                                                                            
    },                                                                                                                                                                                                                                                                                                                         
    "versions": [
        "r0.0.1",
        "r0.1.0",
        "r0.2.0",
        "r0.3.0",
        "r0.4.0"
    ]
}
* Connection #0 to host matrix.port1024.net left intact
  • Install method:

own dockerimage

  • Platform:

docker/linux
@Half-Shot
Copy link
Collaborator

What does your docker image contain (assume own dockerimage means one you've packaged yourself?). Sounds like either you've packaged it with an older/unsupported version of the python library psycopg2 or our own dependency list is wrong.

@silvio
Copy link
Author

silvio commented Apr 22, 2019

Yeah it looks like this was the problem. I updated the psycopg2 to 2.8 and this problem is gone.

But now I have another problem :-(

Apr 22 13:15:49 sfserver docker[19252]: 2019-04-22 11:15:49,363 - synapse.federation.federation_server - 857 - ERROR - PUT-1045- Failed to handle edu u'm.device_list_update'
Apr 22 13:15:49 sfserver docker[19252]: Traceback (most recent call last):
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/federation/federation_server.py", line 853, in on_edu
Apr 22 13:15:49 sfserver docker[19252]:     yield handler(origin, content)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Apr 22 13:15:49 sfserver docker[19252]:     result = result.throwExceptionIntoGenerator(g)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Apr 22 13:15:49 sfserver docker[19252]:     return g.throw(self.type, self.value, self.tb)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/handlers/device.py", line 512, in incoming_device_list_update
Apr 22 13:15:49 sfserver docker[19252]:     yield self._handle_device_updates(user_id)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Apr 22 13:15:49 sfserver docker[19252]:     result = result.throwExceptionIntoGenerator(g)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Apr 22 13:15:49 sfserver docker[19252]:     return g.throw(self.type, self.value, self.tb)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/util/metrics.py", line 63, in measured_func
Apr 22 13:15:49 sfserver docker[19252]:     r = yield func(self, *args, **kwargs)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Apr 22 13:15:49 sfserver docker[19252]:     result = result.throwExceptionIntoGenerator(g)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Apr 22 13:15:49 sfserver docker[19252]:     return g.throw(self.type, self.value, self.tb)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/handlers/device.py", line 596, in _handle_device_updates
Apr 22 13:15:49 sfserver docker[19252]:     user_id, devices, stream_id,
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Apr 22 13:15:49 sfserver docker[19252]:     result = result.throwExceptionIntoGenerator(g)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Apr 22 13:15:49 sfserver docker[19252]:     return g.throw(self.type, self.value, self.tb)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 408, in runInteraction
Apr 22 13:15:49 sfserver docker[19252]:     *args, **kwargs
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Apr 22 13:15:49 sfserver docker[19252]:     result = result.throwExceptionIntoGenerator(g)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Apr 22 13:15:49 sfserver docker[19252]:     return g.throw(self.type, self.value, self.tb)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 457, in runWithConnection
Apr 22 13:15:49 sfserver docker[19252]:     inner_func, *args, **kwargs
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 250, in inContext
Apr 22 13:15:49 sfserver docker[19252]:     result = inContext.theWork()
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
Apr 22 13:15:49 sfserver docker[19252]:     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
Apr 22 13:15:49 sfserver docker[19252]:     return self.currentContext().callWithContext(ctx, func, *args, **kw)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
Apr 22 13:15:49 sfserver docker[19252]:     return func(*args,**kw)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py", line 306, in _runWithConnection
Apr 22 13:15:49 sfserver docker[19252]:     compat.reraise(excValue, excTraceback)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py", line 297, in _runWithConnection
Apr 22 13:15:49 sfserver docker[19252]:     result = func(conn, *args, **kw)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 453, in inner_func
Apr 22 13:15:49 sfserver docker[19252]:     return func(conn, *args, **kwargs)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 329, in _new_transaction
Apr 22 13:15:49 sfserver docker[19252]:     r = func(txn, *args, **kwargs)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/devices.py", line 671, in _update_remote_device_list_cache_txn
Apr 22 13:15:49 sfserver docker[19252]:     lock=False,
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 672, in _simple_upsert_txn
Apr 22 13:15:49 sfserver docker[19252]:     insertion_values=insertion_values,
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 790, in _simple_upsert_txn_native_upsert
Apr 22 13:15:49 sfserver docker[19252]:     txn.execute(sql, list(allvalues.values()))
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 119, in execute
Apr 22 13:15:49 sfserver docker[19252]:     self._do_execute(self.txn.execute, sql, *args)
Apr 22 13:15:49 sfserver docker[19252]:   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 149, in _do_execute
Apr 22 13:15:49 sfserver docker[19252]:     sql, *args
Apr 22 13:15:49 sfserver docker[19252]: InvalidColumnReference: there is no unique or exclusion constraint matching the ON CONFLICT specification

Its better to open a new issue for this?

@silvio
Copy link
Author

silvio commented Apr 22, 2019

This problem can be fixed with this comment

@silvio silvio closed this as completed Apr 22, 2019
@Nebukadneza Nebukadneza mentioned this issue May 4, 2019
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@silvio @Half-Shot