Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

missing pyisam.backend.cffi._isam_cffi #1

Open
qippur opened this issue Dec 29, 2023 · 29 comments
Open

missing pyisam.backend.cffi._isam_cffi #1

qippur opened this issue Dec 29, 2023 · 29 comments
Assignees
@rpmoseley

Comments

@qippur
Copy link

qippur commented Dec 29, 2023

Hi, I know this repo is not updated since a lot, but I am trying anyway to ask.

I cloned the repo, compiled vbsam and tried to run some of the utils.
I am stuck at this:

pyisam$ python3 run_wing.py
.
.
.
ModuleNotFoundError: No module named 'pyisam.backend.cffi._isam_cffi'

Could you point me to what to do to fix this issue?

Thanks in advance
@rpmoseley
Copy link
Owner

rpmoseley commented Dec 29, 2023 via email

@qippur
Copy link
Author

qippur commented Dec 29, 2023 via email

@qippur
Copy link
Author

qippur commented Dec 30, 2023 via email

@qippur
Copy link
Author

qippur commented Dec 30, 2023 via email

@rpmoseley
Copy link
Owner

Hi,
I have updated the repository to put back the dumprec module in pyisam, which I had moved into the tests/ directory as part of moving away from having the tests within the main package. If you the 'python -m pyisam -h' you should get a help message giving the available tests. If you then run 'python -m pyisam -t12', and you have the sample data available under data/ you should get the following:

TEST: 12
ASSUME: TableIndex(name='comp', dups=False, desc=False, colinfo=[TableIndexCol(name='comp')])
FILL_BEF: Record_utool_dec_decomp(comp='', comptyp='', sys='', prefix='', user='', database='', release='', timeup=0, specup=0)
Record_utool_dec_decomp(comp='defile', comptyp='F', sys='mx', prefix='def', user='adf', database='utool', release='Y', timeup=820246079, specup=0)

I am working on getting the support for vbisam added to the package, and will push it when ready.

However, here are the details for building the extension, though you seemed to have done this already. From a fresh repository with no _isam_cffi support under pyisam/backend/cffi, running the following command from the root directory of the repository will generate the module for the version of python you are running:

$ python utils/isam_build.py

You will then find a _isam_cffi*.so file in pyisam/backend/cffi, this should depend on libifisam.so and libifisamx.so, which should have been copied into pyisam/backend/lib. You can check the dependencies using 'ldd' or 'patchelf --print-needed' if installed.

Richard

@rpmoseley
Copy link
Owner

Hi, further to my last comment, if you wish to switch the backend to ctypes, edit the file pyisam/backend/init.py and change the variable 'conf' to "'ctypes'". This will change to be overridable with the environment variable 'PYISAM_BACKEND' once I've pushed the latest changes.

@qippur
Copy link
Author

qippur commented Dec 30, 2023 via email

@rpmoseley
Copy link
Owner

There was a problem in the 'utils/isam_build.py' where the definition of di_nrecords was long, which is 64 bits, CISAM expects an int (32 bits). I have pushed a change to fix the problem, alternately you can change the line 79 from:
long di_nrecords
to:
{longsz} di_nrecords.
(Note the brackets are required, as the type of field is changed if your running on a 32-bit architecture).

@qippur
Copy link
Author

qippur commented Dec 30, 2023 via email

@rpmoseley
Copy link
Owner

Oops!! I managed to loose them, I have now re-added them, I left a .gitignore file that meant that they were not sent. Now fixed.

@qippur
Copy link
Author

qippur commented Dec 30, 2023 via email

@rpmoseley
Copy link
Owner

Could you send me the tar file that you mentioned, as it has been removed from the comments, my email address is:

richard.moseley4@gmail.com

I can then take a look.

First impressions would imply that you have no keys defined on the table, that is zero. Thus asking for the first key will fail with the error that you gave.

@qippur
Copy link
Author

qippur commented Dec 30, 2023 via email

@rpmoseley
Copy link
Owner

Your problem is that your adding one to the index number, then the pyisam also adds one to that. Thus when using the pyisam library, the first key is 0, which maps to key 1 to the ISAM iskeyinfo() function. I will send you the source code to a program which performs an also identical action as your lstisam.py. I will send you a tarball with my version of the script, which should be placed into the root directory of the environment, the script expects the data to be under exdata, but change that to suit.

@qippur
Copy link
Author

qippur commented Dec 30, 2023 via email

@rpmoseley
Copy link
Owner

I have now pushed my changes which will now pull a version of pyisam which can be used within a virtualenv using the script 'mkvenv' and support both CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared libraries in the repository, so that you no longer need to compile the extension modules before use. The README.md file has been updated with more useful information. According to the README that is part of the CISAM distribution, the use of isindexinfo() is deprecated due to the fact that the second argument is a void *, and there are now two separate functions, isdictinfo() which is the same as calling isindexinfo() with a key of zero, and iskeyinfo() which is the same as calling if isindexinfo() is called with a key of non-zero. I trust this will help you continue your work.

@qippur
Copy link
Author

qippur commented Jan 3, 2024 via email

@qippur
Copy link
Author

qippur commented Jan 4, 2024 via email

@rpmoseley
Copy link
Owner

Hi, I can do that, if you could let me know how the .fld file stores the field information.

I noticed that the first index on the example table has 0 key parts, this indicates that the table can be referenced using record numbers. The program 'isql' (which was a front-end for Informix, stored information about tables in tables named 'sys*', the field information was stored in 'syscolumns'.

vbisam seems to always return zero rows when you request the dictinfo, seems that there may be a speed issue, as the original code called a function, tcountrows. Looking at the comments, it would appear that the tnrows is set to 0 or 1 depending on whether or not there are any records in the table, I assume that this is because, the program using the library is expected to perform the count itself. This will need some work to correct, as I have compiled the support in and now it gives a count of 65166 against 807 that CISAM gives, there must something else going there.

As for an example, I assume that you want an example written in python, I will work on it and get back to you once finished. At the sametime I will look into what the isdictino() function of CISAM actually calls.

Richard

@rpmoseley
Copy link
Owner

rpmoseley commented Jan 5, 2024 via email

@qippur
Copy link
Author

qippur commented Jan 5, 2024 via email

@qippur
Copy link
Author

qippur commented Jan 5, 2024 via email

@rpmoseley
Copy link
Owner

Roberto,

Yes the image has been removed from the comments, could you send it directly to me at richard.moseley4@gmail.com, in the meantime, I've updated my additional isdictinfo() now uses a local copy of countrows() which now returns the correct row count, which I have now pushed. I am in the process of checking the difference between 32-bit and 64-bit version of the underlying ISAM formats.

Richard

@rpmoseley
Copy link
Owner

I've now written a module called 'pyisam.tabdefns.fldfile' which will take one of the .fld files and generate the table definition from it, using a DynamicTableDefn. Since there is no field names or index information within the file, the module generates fields of the format 'fldNNNN' where the first field is numbered 1. The pyisam package will attempt to match the indexes when the first read is performed. I'm still working on a complete program that creates a new table, inserts data, updates data and deletes data, and finally removes the table again.

@qippur
Copy link
Author

qippur commented Jan 11, 2024 via email

@rpmoseley
Copy link
Owner

Yes, its quite simple, each element is 4 bytes stored LSB (for example 0x08 0x00 0x00 0x00), the first such element is the number of fields that follow, then for each field the first element is the type of data, and the the second seems to be screen size, which in combination with the image of the table you gave me, allowed me guess suitable types of field. If you look at _fldtype_map in the fldfile.py module this indicates that they are not all types are mapped. It is then a simple case of adding suitable TableColumn instances to the definition. There must be somewhere else that the field and index information is stored, as there is no field name or indexes within the .fld file.

@rpmoseley
Copy link
Owner

I do apologise for not getting back sooner, but as I was writing the missing support for inserting, deleting and updating records, I've encountered an issue over the creation of the underlying record buffer, which is present in the layer above the ISAMobject which is causing me a headache.

@qippur
Copy link
Author

qippur commented Feb 9, 2024 via email

@rpmoseley
Copy link
Owner

I have just made a push of code that ensures that the libvbisam library gets initialised correctly. Previously it was using data as left by the operating system, and happened to leave the file descriptor for the logging to zero, which when the first (or only) table was opened, the index file was opened and given the same file descriptor and hence all the logging information gets appended to the end of the index file incorrectly. The fix is to make the vbisam backend call the appropriate routine to establish the file descriptors as -1 to disable the logging by default.

@rpmoseley rpmoseley self-assigned this Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rpmoseley rpmoseley

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rpmoseley @qippur