Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

core dumped when logger level = 10 #62

Closed
wjaskowski opened this issue Oct 10, 2019 · 4 comments
Closed

core dumped when logger level = 10 #62

wjaskowski opened this issue Oct 10, 2019 · 4 comments
Assignees
@pitercl @kamil-kaczmarek

Comments

@wjaskowski
Copy link

Context: SacredObserver but probably related to Neptune itself.

Whenever I use logger level of 10, I get core dumped error. logger level of >=20 works fine. Stacktrace:

(...)
I1010 13:52:19.454688 140211946456832 connectionpool.py:393] https://ui.neptune.ml:443 "POST /api/backend/v1/channels/system?experimentId=26f075d2-2412-4971-81c9-c5322745a03c HTTP/1.1" 200 82
I1010 13:52:19.455582 140211946456832 client.py:239] createSystemChannel({'experimentId': '26f075d2-2412-4971-81c9-c5322745a03c', 'channelToCreate': ChannelParams(channelType='text', name='stderr')})
I1010 13:52:19.456417 140211946456832 client.py:239] createSystemChannel({'experimentId': '26f075d2-2412-4971-81c9-c5322745a03c', 'channelToCreate': ChannelParams(channelType='text', name='stderr')})
Fatal Python error: Cannot recover from stack overflow.

Thread 0x00007f852299f700 (most recent call first):
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 300 in wait
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/queue.py", line 179 in get
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 111 in run
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 917 in _bootstrap_inner
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 885 in _bootstrap

Current thread 0x00007f85a3430700 (most recent call first):
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/posixpath.py", line 42 in _get_sep
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/posixpath.py", line 147 in basename
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 327 in __init__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1488 in makeRecord
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1518 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/urllib3/connectionpool.py", line 393 in _make_request
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600 in urlopen
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/requests_client.py", line 392 in result
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/http_future.py", line 290 in _get_incoming_response
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/http_future.py", line 124 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/http_future.py", line 197 in response
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  ...

Thread 0x00007f85a2c2f700 (most recent call first):
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/ssl.py", line 913 in read
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/ssl.py", line 1037 in recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_socket.py", line 84 in _recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_socket.py", line 102 in recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 452 in _recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_abnf.py", line 396 in recv_strict
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_abnf.py", line 309 in recv_header
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_abnf.py", line 361 in recv_frame
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 377 in recv_frame
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 343 in recv_data_frame
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 330 in recv_data
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/websockets/websocket_client_adapter.py", line 34 in recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/websockets/reconnecting_websocket.py", line 43 in recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/threads/aborting_thread.py", line 34 in run
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 917 in _bootstrap_inner
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f85a7844740 (most recent call first):
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 854 in acquire
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 903 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 585 in create_hardware_metric
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/hardware/metrics/service/metric_service_factory.py", line 40 in create
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/execution/execution_context.py", line 151 in _run_hardware_metrics_reporting_thread
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/execution/execution_context.py", line 87 in start
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 987 in _start
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/projects.py", line 431 in create_experiment
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/__init__.py", line 230 in create_experiment
  File "/home/wojciech/projects/magellan/sacred_utils.py", line 134 in started_event
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/run.py", line 326 in _emit_started
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/run.py", line 230 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/experiment.py", line 212 in run
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/experiment.py", line 263 in run_commandline
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/experiment.py", line 140 in automain
  File "main.py", line 716 in <module>
[1]    10623 abort (core dumped)  main.py
@wjaskowski
Copy link
Author

Btw. any ideas how can I filter out those neptune (?) logger messages at logger.level=10?

@kamil-kaczmarek kamil-kaczmarek self-assigned this Oct 12, 2019
@kamil-kaczmarek
Copy link
Contributor

Hey @wjaskowski, good question. At this point I do not know (regarding neptune logger massages).

@pitercl, @jakubczakon - do you guys have better ideas?

@pitercl
Copy link
Member

pitercl commented Oct 14, 2019

@kamil-kaczmarek, I'll take a look!

@kamil-kaczmarek
Copy link
Contributor

@pitercl thx!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pitercl pitercl

@kamil-kaczmarek kamil-kaczmarek

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wjaskowski @pitercl @kamil-kaczmarek