Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detailed error in case of S3's libcurl network failure #1265

Merged
merged 2 commits into from
Jan 25, 2024

Conversation

IvoDD
Copy link
Collaborator

@IvoDD IvoDD commented Jan 23, 2024

Reference Issues/PRs

Closes #880

What does this implement or fix?

Previously on network connection failures we'd get an error message like Got unexpected error: '99' : curlCode: 6, Couldn't resolve host name.

Now we explain that this can be either caused by connection issues or by exhausted open sockets and suggests increasing ulimit -n.

Any other comments?

Didn't add a unit test since it's hard to reproduce in a unit test environment. So I tested locally as follows:

$ cat many_arctic.py
from arcticdb import Arctic

aas = []

for i in range(1000):
    print(i)
    arctic = Arctic("s3://172.22.107.88:test?port=9000&access=<ACCESS>&secret=<SECRET>")
    aas.append(arctic)
    arctic.has_library(f"lib_{i}")
$ ulimit -n 50
$ python many_arctic.py
0
1
2
3
4
5
6
7
8
[2024-01-23 16:35:53.717] [arcticdb] [error] Got an unexpected network error: curlCode: 7, Couldn't connect to server.
This could be due to a connectivity issue or exhausted open sockets. In case socket file descriptors are exhausted, consider increasing `ulimit -n`.
Traceback (most recent call last):
  File "/home/ivo/source/ArcticDB/python/many_arctic.py", line 9, in <module>
    arctic.has_library(f"lib_{i}")
  File "/home/ivo/source/ArcticDB/python/arcticdb/arctic.py", line 345, in has_library
    return self._library_manager.has_library(self._library_adapter.get_name_for_library_manager(name))
arcticdb_ext.exceptions.InternalException: arcticdb::storage::s3::detail::UnexpectedS3ErrorException(Got an unexpected network error: curlCode: 7, Couldn't connect to server. This could be due to a connectivity issue or too many open Arctic instances. Having more than one open Arctic instance is not advised, you should reuse them. If you absolutely need many open Arctic instances, consider increasing `ulimit -n`.)

Checklist

Checklist for code changes...
  • Have you updated the relevant docstrings, documentation and copyright notice?
  • Is this contribution tested against all ArcticDB's features?
  • Do all exceptions introduced raise appropriate error messages?
  • Are API changes highlighted in the PR description?
  • Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

@IvoDD IvoDD self-assigned this Jan 23, 2024
@IvoDD IvoDD marked this pull request as ready for review January 23, 2024 14:38
@IvoDD IvoDD requested a review from poodlewars January 23, 2024 14:38
cpp/arcticdb/storage/s3/detail-inl.hpp Outdated Show resolved Hide resolved
NETWORK_CONNECTION libcurl errors can be caused because of internet
issues and exhausted socket file descriptors.

Now we convey this information with a more detailed log instead of the
confusing "libcurl code: 7"
@IvoDD IvoDD merged commit 06a52de into master Jan 25, 2024
130 checks passed
@IvoDD IvoDD deleted the fix-many-arctic-instances branch January 25, 2024 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

libcurl error when executing requests across large numbers of Arctic and Library instances in python
2 participants