I encountered an error that make transfer crash with a too long download duration, caused by a duration too big for that Integer field. Solved by using BigInteger type.
Traceback (most recent call last):
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/apscheduler/executors/base.py", line 125, in run_job
retval = job.func(*job.args, **job.kwargs)
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/export_vn/transfer_vn.py", line 402, in full_download_1
short_version=(1 if cfg.json_format == "short" else 0),
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/export_vn/download_vn.py", line 665, in store
taxo, territorial_unit_ids, short_version=short_version
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/export_vn/download_vn.py", line 587, in _store_search
timing,
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/export_vn/store_all.py", line 132, in log
site, controler, error_count, http_status, comment, length, duration
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/export_vn/store_postgresql.py", line 1259, in log
self._conn.execute(stmt)
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1262, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
self, multiparams, params, execution_options
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1461, in _execute_clauseelement
cache_hit=cache_hit,
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1814, in _execute_context
e, statement, parameters, cursor, context
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1995, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context
cursor, statement, parameters, context
File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.errors.NumericValueOutOfRange) integer out of range
[SQL: INSERT INTO src_vn_json.download_log (site, controler, error_count, http_status, comment, length, duration) VALUES (%(site)s, %(controler)s, %(error_count)s, %(http_status)s, %(comment)s, %(length)s, %(duration)s) RETURNING src_vn_json.download_log.id]
[parameters: {'site': 'ff', 'controler': 'observations', 'error_count': 8, 'http_status': 200, 'comment': 'ff => Iter: 20, 2139 obs, taxo_group: 1, territorial_unit: 53, date: 24/04/2021, interval: 5', 'length': 8405913, 'duration': 3649730952.082}]
(Background on this error at: http://sqlalche.me/e/14/9h9h)
2021-10-20 21:59:59,599 - ERROR - transfer_vn - The job Observations_ff crashed
In GitLab by @fred.lpo07 on Oct 21, 2021, 11:37
I encountered an error that make transfer crash with a too long download duration, caused by a duration too big for that Integer field. Solved by using BigInteger type.