[infinity] infinity crashes

[nainamd77]

Hi,
We have creating set of tables(delete if exists and create) and loading them with data using set of methods. These methods called in a constructor.
But when it is called frequently i.e. delete, create and data load , infinity service crashed with following error messages.
Then we need to restart the infinity again.

Exception Message:
terminate called after throwing an instance of 'irs::index_not_found'
what(): No segments* file found.
Aborted (core dumped)

16:50:05.854] [19832] [critical] WalManager::Checkpoint failed: Can't open file: /tmp/infinity/data/catalog/META_CATALOG.21006.json: Too many open files@src/storage/io/local_file_system.cpp:84
terminate called after throwing an instance of 'infinity::UnrecoverableException@infinity_exception'
what(): Can't open file: /tmp/infinity/data/catalog/META_CATALOG.21006.json: Too many open files@src/storage/io/local_file_system.cpp:84
Aborted (core dumped)

code files:
manage_class.txt
processreq_t.txt

infinity info
infinity-sdk 0.1.0.dev7
infinity 0.1.0 build on 2024-02-26 00:50

below the snippets from the attached code files

class manage_class:
def init(self):
# Initialize the infinity connection and database at the time of class instantiation
self.infinity_obj = infinity.connect(REMOTE_HOST)
self.db = self.infinity_obj.get_database("default")
self.LoadClassStatesMaster()
self.LoadClassDetails()
self.LoadClassActionsMaster()
self.LoadMasterData()
self.LoadIntentData()

->exception thrown between any of load* methods

Above class instantiated and used in below class

class proces:
def process_data(request):
global generator # Declare 'generator' as a global variable
j_response = None
obj_mcs = manage_class.manage_class()

above classes part of django web api application, so frequent http requests hit the application.

To avoid crashing, We have implemented a work around using a flag check to ensure that table creation and data loading methods called only once.

• Also anyway to find whether a class already exists while creating?

[JinHai-CN]

1. For 'Too many open files', you may try to use 'ulimit' command to increase the maximum number of files that the calling process can open.
2. By running the file you attached, we also find some problems of infinity. Once the problems are resolved, we will let you know.

[nainamd77]

hi, any update on this ?

[JinHai-CN]

hi, any update on this ?

Most of the issues are resolved. And we are planning to release the first stable version in 2 or 3 weeks. I will let you know when it is ready.

[nainamd77]

good. thanks for info.

[nainamd77]

just to report, on 24-04-24 I downloaded latest release of the date and trying to do some DML operations, but it crashed. Now I need to reinstall all to work again.
Name: infinity-sdk
Version: 0.1.0.dev19

[12:16:51.057] [6968] [info] Read delta op: AddTableEntryOp type: ADD_TABLE_ENTRY, begin_ts: 32251, txn_id: 3736, commit_ts: 41828, merge_flag: 3 db_name: default table_name: t_webintent_data table_entry_dir: /var/infinity/data/IlhVWG9jcI_db_default/Rg3LqOXCXd_table_t_webintent_data column_def: model_name Varchar column_def: initial_intent Varchar column_def: sub_intent1 Varchar column_def: sub_intent2 Varchar column_def: refined_intent Varchar column_def: cdate Varchar column_def: lmdate Varchar row_count: 23 unsealed_id: 0 next_segment_id: 1
[12:16:51.057] [6968] [info] Read delta op: AddTableEntryOp type: ADD_TABLE_ENTRY, begin_ts: 269, txn_id: 135, commit_ts: 41828, merge_flag: 3 db_name: default table_name: t_webscrape_data table_entry_dir: /var/infinity/data/IlhVWG9jcI_db_default/UoJB2G81FH_table_t_webscrape_data column_def: model_name Varchar column_def: id Varchar column_def: parent_id Varchar column_def: intent Varchar column_def: intent_vec Embedding column_def: text Varchar column_def: text_vec Embedding row_count: 29 unsealed_id: 0 next_segment_id: 1
[12:16:51.057] [6968] [info] Read delta op: AddTableEntryOp type: ADD_TABLE_ENTRY, begin_ts: 34421, txn_id: 4821, commit_ts: 41828, merge_flag: 3 db_name: default table_name: user_session_table table_entry_dir: /var/infinity/data/IlhVWG9jcI_db_default/hyWsmpxzQN_table_user_session_table column_def: user_id Varchar column_def: session_id Varchar column_def: user_query Varchar column_def: response Varchar column_def: options Varchar column_def: data Varchar column_def: active Integer column_def: cdate Varchar column_def: lmdate Varchar row_count: 2 unsealed_id: 0 next_segment_id: 1
[12:16:51.057] [6968] [info] Replay phase 3: replay 1 entries
terminate called after throwing an instance of 'infinity::UnrecoverableException@infinity_exception'
what(): Decrease row count exceed actual row count@src/storage/meta/entry/segment_entry.cppm:184

[nainamd77]

another issue, today 26-apr-24 i downloaded and installed latest infinity along with latest sdk infinity-sdk-0.1.0.dev21, When I trying to create a table i got following error 'string indices must be integers'
below full exception

Exception ignored in: <function RemoteThriftInfinityConnection.del at 0x7fb8f18fd120>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/infinity/remote_thrift/infinity.py", line 36, in del
self.disconnect()
File "/usr/local/lib/python3.10/dist-packages/infinity/remote_thrift/infinity.py", line 96, in disconnect
res = self._client.disconnect()
File "/usr/local/lib/python3.10/dist-packages/infinity/remote_thrift/client.py", line 200, in disconnect
res = self.client.Disconnect(CommonRequest(session_id=self.session_id))
File "/usr/local/lib/python3.10/dist-packages/infinity/remote_thrift/infinity_thrift_rpc/InfinityService.py", line 305, in Disconnect
self.send_Disconnect(request)
File "/usr/local/lib/python3.10/dist-packages/infinity/remote_thrift/infinity_thrift_rpc/InfinityService.py", line 309, in send_Disconnect
self._oprot.writeMessageBegin('Disconnect', TMessageType.CALL, self._seqid)
File "/home/barrla/.local/lib/python3.10/site-packages/thrift/protocol/TBinaryProtocol.py", line 54, in writeMessageBegin
self.writeI32(TBinaryProtocol.VERSION_1 | type)
File "/home/barrla/.local/lib/python3.10/site-packages/thrift/protocol/TBinaryProtocol.py", line 119, in writeI32
self.trans.write(buff)
File "/home/barrla/.local/lib/python3.10/site-packages/thrift/transport/TTransport.py", line 173, in write
raise e
File "/home/barrla/.local/lib/python3.10/site-packages/thrift/transport/TTransport.py", line 169, in write
self.__wbuf.write(buf)
ValueError: I/O operation on closed file.

Code:
self.infinity_obj = infinity.connect(REMOTE_HOST)
self.db = self.infinity_obj.get_database("default")

self.db.create_table("t_webscrape_data", {
"model_name": "varchar",
"id": "varchar",
"parent_id": "varchar",
"intent": "varchar",
"intent_vec": "vector, 4, float",
"text": "varchar",
"text_vec": "vector, 4, float"
})

[JinHai-CN]

@nainamd77

0.1.0 is released which is much more stable. You can use this version.

[JinHai-CN]

In the latest version, the default database name is 'default_db'.

[JinHai-CN]

Use following code to create table:

self.db.create_table("t_webscrape_data", {
"model_name": {"type": "varchar"},
"id": {"type": "varchar"},
"parent_id": {"type": "varchar"},
"intent": {"type": "varchar"},
"intent_vec": {"type": "vector, 4, float"},
"text": {"type": "varchar"},
"text_vec": {"type": "vector, 4, float"}
})