-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
missing pyisam.backend.cffi._isam_cffi #1
Comments
Hi, I thank you for cloning this, I realise that I have not pushed anything
in sometime. I've recently been updating my local copy.
The run_wing.py script is there to be able to debug within the WingIDE
(which I assume you're doing). Otherwise, if you execute, 'python3 -m
pyisam -h' you will get a list of arguments and the current tests that can
be run.
The missing library 'pyisam.backend.cffi._isam_cffi' is generated by the
script 'bldlibisam.py'. You can switch to using ctypes if you edit the
__init__.py file and change the value of use_cffi to False. In order to
use the 'ctypes' backend, you need to create a library named 'libpyisam'
that needs the libvbisam library.
The version of the repository has it stands has a script named
'isam_build.py' within the 'utils' subdirectory which will build the
necessary extension using the appropriate libraries, though, it has not yet
been tested using the libvbisam library as I had an issue with it when I
first started.
I will aim to push a change before the end of the year with the latest
changes once I've updated my master copy.
There has been some re-organisation of the source tree, and there is now a
separate configuration file. I will also push both the 32- and 64-bit
original Informix (now IBM) C-ISAM libraries as well, as they are required
to run programs linked to use the CISAM support.
Richard Moseley
…On Fri, 29 Dec 2023, 11:28 qippur, ***@***.***> wrote:
Hi, I know this repo is not updated since a lot, but I am trying anyway to
ask.
I cloned the repo, compiled vbsam and tried to run some of the utils.
I am stuck at this:
pyisam$ python3 run_wing.py
.
.
.
ModuleNotFoundError: No module named 'pyisam.backend.cffi._isam_cffi'
Could you point me to what to do to fix this issue?
Thanks in advance
—
Reply to this email directly, view it on GitHub
<#1>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB6UJZFVZWPC7ARLUXFDRADYL2SMLAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TSMZVGU4TKOA>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Hi Richard,
Thank you for your quick response!
I will wait for your update and then try to follow your directions.
I will try with both vbisam (preferred for licensing issues) and IBM
libraries once they are pushed.
I am more proficient in using the libraries in python than in preparing
them for usage, but I'll do my best.
Thanks again, and happy new year!
roberto resecco
Il giorno ven 29 dic 2023 alle ore 17:32 Richard Moseley <
***@***.***> ha scritto:
… Hi, I thank you for cloning this, I realise that I have not pushed
anything
in sometime. I've recently been updating my local copy.
The run_wing.py script is there to be able to debug within the WingIDE
(which I assume you're doing). Otherwise, if you execute, 'python3 -m
pyisam -h' you will get a list of arguments and the current tests that can
be run.
The missing library 'pyisam.backend.cffi._isam_cffi' is generated by the
script 'bldlibisam.py'. You can switch to using ctypes if you edit the
__init__.py file and change the value of use_cffi to False. In order to
use the 'ctypes' backend, you need to create a library named 'libpyisam'
that needs the libvbisam library.
The version of the repository has it stands has a script named
'isam_build.py' within the 'utils' subdirectory which will build the
necessary extension using the appropriate libraries, though, it has not
yet
been tested using the libvbisam library as I had an issue with it when I
first started.
I will aim to push a change before the end of the year with the latest
changes once I've updated my master copy.
There has been some re-organisation of the source tree, and there is now a
separate configuration file. I will also push both the 32- and 64-bit
original Informix (now IBM) C-ISAM libraries as well, as they are required
to run programs linked to use the CISAM support.
Richard Moseley
On Fri, 29 Dec 2023, 11:28 qippur, ***@***.***> wrote:
> Hi, I know this repo is not updated since a lot, but I am trying anyway
to
> ask.
>
> I cloned the repo, compiled vbsam and tried to run some of the utils.
> I am stuck at this:
>
> pyisam$ python3 run_wing.py
> .
> .
> .
> ModuleNotFoundError: No module named 'pyisam.backend.cffi._isam_cffi'
>
> Could you point me to what to do to fix this issue?
>
> Thanks in advance
>
> —
> Reply to this email directly, view it on GitHub
> <#1>, or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AB6UJZFVZWPC7ARLUXFDRADYL2SMLAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TSMZVGU4TKOA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUKB3JPVLXPTT45YZATYL3WABAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGIYDKMRQGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Sorry to bother, I cannot find the script bldlibisam.py
What am I missing?
Roberto
…--
Il Ven 29 Dic 2023, 17:32 Richard Moseley ***@***.***> ha
scritto:
Hi, I thank you for cloning this, I realise that I have not pushed
anything
in sometime. I've recently been updating my local copy.
The run_wing.py script is there to be able to debug within the WingIDE
(which I assume you're doing). Otherwise, if you execute, 'python3 -m
pyisam -h' you will get a list of arguments and the current tests that can
be run.
The missing library 'pyisam.backend.cffi._isam_cffi' is generated by the
script 'bldlibisam.py'. You can switch to using ctypes if you edit the
__init__.py file and change the value of use_cffi to False. In order to
use the 'ctypes' backend, you need to create a library named 'libpyisam'
that needs the libvbisam library.
The version of the repository has it stands has a script named
'isam_build.py' within the 'utils' subdirectory which will build the
necessary extension using the appropriate libraries, though, it has not
yet
been tested using the libvbisam library as I had an issue with it when I
first started.
I will aim to push a change before the end of the year with the latest
changes once I've updated my master copy.
There has been some re-organisation of the source tree, and there is now a
separate configuration file. I will also push both the 32- and 64-bit
original Informix (now IBM) C-ISAM libraries as well, as they are required
to run programs linked to use the CISAM support.
Richard Moseley
On Fri, 29 Dec 2023, 11:28 qippur, ***@***.***> wrote:
> Hi, I know this repo is not updated since a lot, but I am trying anyway
to
> ask.
>
> I cloned the repo, compiled vbsam and tried to run some of the utils.
> I am stuck at this:
>
> pyisam$ python3 run_wing.py
> .
> .
> .
> ModuleNotFoundError: No module named 'pyisam.backend.cffi._isam_cffi'
>
> Could you point me to what to do to fix this issue?
>
> Thanks in advance
>
> —
> Reply to this email directly, view it on GitHub
> <#1>, or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AB6UJZFVZWPC7ARLUXFDRADYL2SMLAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TSMZVGU4TKOA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUKB3JPVLXPTT45YZATYL3WABAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGIYDKMRQGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Hi Richard,
I created a pipenv virtual environment and I was able to build the library
$ ll ./libisam/64/libpyisam.so
-rwxr-xr-x 1 roberto roberto 9064 30 dic 12.07 ./libisam/64/libpyisam.so
but it seems that some part of pyisam is missing:
$ python3 -m pyisam -h
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/roberto/devel/python/pyisam/pyisam/__main__.py", line 6, in
<module>
from .dumprec import dump_record_imp, dump_record_exp_eq
*ModuleNotFoundError: No module named 'pyisam.dumprec'*
Do you have any suggestions?
roberto resecco
Il giorno sab 30 dic 2023 alle ore 10:52 Roberto Resecco <
***@***.***> ha scritto:
… Sorry to bother, I cannot find the script bldlibisam.py
What am I missing?
Roberto
--
Il Ven 29 Dic 2023, 17:32 Richard Moseley ***@***.***> ha
scritto:
> Hi, I thank you for cloning this, I realise that I have not pushed
> anything
> in sometime. I've recently been updating my local copy.
>
> The run_wing.py script is there to be able to debug within the WingIDE
> (which I assume you're doing). Otherwise, if you execute, 'python3 -m
> pyisam -h' you will get a list of arguments and the current tests that
> can
> be run.
>
> The missing library 'pyisam.backend.cffi._isam_cffi' is generated by the
> script 'bldlibisam.py'. You can switch to using ctypes if you edit the
> __init__.py file and change the value of use_cffi to False. In order to
> use the 'ctypes' backend, you need to create a library named 'libpyisam'
> that needs the libvbisam library.
>
> The version of the repository has it stands has a script named
> 'isam_build.py' within the 'utils' subdirectory which will build the
> necessary extension using the appropriate libraries, though, it has not
> yet
> been tested using the libvbisam library as I had an issue with it when I
> first started.
>
> I will aim to push a change before the end of the year with the latest
> changes once I've updated my master copy.
>
> There has been some re-organisation of the source tree, and there is now
> a
> separate configuration file. I will also push both the 32- and 64-bit
> original Informix (now IBM) C-ISAM libraries as well, as they are
> required
> to run programs linked to use the CISAM support.
>
> Richard Moseley
>
> On Fri, 29 Dec 2023, 11:28 qippur, ***@***.***> wrote:
>
> > Hi, I know this repo is not updated since a lot, but I am trying anyway
> to
> > ask.
> >
> > I cloned the repo, compiled vbsam and tried to run some of the utils.
> > I am stuck at this:
> >
> > pyisam$ python3 run_wing.py
> > .
> > .
> > .
> > ModuleNotFoundError: No module named 'pyisam.backend.cffi._isam_cffi'
> >
> > Could you point me to what to do to fix this issue?
> >
> > Thanks in advance
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <#1>, or unsubscribe
> > <
> https://github.com/notifications/unsubscribe-auth/AB6UJZFVZWPC7ARLUXFDRADYL2SMLAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TSMZVGU4TKOA>
>
> > .
> > You are receiving this because you are subscribed to this
> thread.Message
> > ID: ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <#1 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACXMXUKB3JPVLXPTT45YZATYL3WABAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGIYDKMRQGA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
|
Hi, TEST: 12 I am working on getting the support for vbisam added to the package, and will push it when ready. However, here are the details for building the extension, though you seemed to have done this already. From a fresh repository with no _isam_cffi support under pyisam/backend/cffi, running the following command from the root directory of the repository will generate the module for the version of python you are running: $ python utils/isam_build.py You will then find a _isam_cffi*.so file in pyisam/backend/cffi, this should depend on libifisam.so and libifisamx.so, which should have been copied into pyisam/backend/lib. You can check the dependencies using 'ldd' or 'patchelf --print-needed' if installed. Richard |
Hi, further to my last comment, if you wish to switch the backend to ctypes, edit the file pyisam/backend/init.py and change the variable 'conf' to "'ctypes'". This will change to be overridable with the environment variable 'PYISAM_BACKEND' once I've pushed the latest changes. |
Thank you Richard,
the test ran successfully.
$ python3 -m pyisam -t 12
TEST: 12
ASSUME: TableIndex(name='comp', dups=False, desc=False,
colinfo=[TableIndexCol(name='comp')])
FILL_BEF: Record_utool_dec_decomp(comp='', comptyp='', sys='', prefix='',
user='', database='', release='', timeup=0, specup=0)
Record_utool_dec_decomp(comp='defile', comptyp='F', sys='mx', prefix='def',
user='adf', database='utool', release='Y', timeup=820246079, specup=0)
Then I tried to open some C-ISAM tables (attached) created with IBM C-ISAM
libraries. I created a "tables" folder, copied my tables there, placed and
edited the util lstisam.py, but I get this:
$ python3 tables/lstisam.py
Traceback (most recent call last):
File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 37, in
<module>
lstisam('cgtotst1', 'tables')
File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 31, in
lstisam
ti, ii = fetch_isam_info(tabname, tabpath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 17, in
fetch_isam_info
tabinfo = tabobj.isdictinfo()
^^^^^^^^^^^^^^^^^^^
File "/home/roberto/devel/python/pyisam/pyisam/backend/cffi/__init__.py",
line 311, in isdictinfo
dinfo = ffi.new('struct dictinfo *')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ffi.error: struct dictinfo: wrong size for field 'di_nrecords' (cdef says
8, but C compiler says 4). fix it or use "...;" as the last field in the
cdef for struct dictinfo to make it flexible
The .fld file in the tar is a field descriptor text file used by the
language to map the fields in the table.
As you probably understood, the underlying C stuff is not my bread and
butter.
I found the 2.1.1 version of vbisam HERE
<https://www.arnoldtrembley.com/GnuCOBOL.htm>. It compiles a
libvbisam.so.3.0.0 file.
roberto
Il giorno sab 30 dic 2023 alle ore 15:21 Richard Moseley <
***@***.***> ha scritto:
… Hi,
I have updated the repository to put back the dumprec module in pyisam,
which I had moved into the tests/ directory as part of moving away from
having the tests within the main package. If you the 'python -m pyisam -h'
you should get a help message giving the available tests. If you then run
'python -m pyisam -t12', and you have the sample data available under data/
you should get the following:
TEST: 12
ASSUME: TableIndex(name='comp', dups=False, desc=False,
colinfo=[TableIndexCol(name='comp')])
FILL_BEF: Record_utool_dec_decomp(comp='', comptyp='', sys='', prefix='',
user='', database='', release='', timeup=0, specup=0)
Record_utool_dec_decomp(comp='defile', comptyp='F', sys='mx',
prefix='def', user='adf', database='utool', release='Y', timeup=820246079,
specup=0)
I am working on getting the support for vbisam added to the package, and
will push it when ready.
However, here are the details for building the extension, though you
seemed to have done this already. From a fresh repository with no
_isam_cffi support under pyisam/backend/cffi, running the following command
from the root directory of the repository will generate the module for the
version of python you are running:
$ python utils/isam_build.py
You will then find a _isam_cffi*.so file in pyisam/backend/cffi, this
should depend on libifisam.so and libifisamx.so, which should have been
copied into pyisam/backend/lib. You can check the dependencies using 'ldd'
or 'patchelf --print-needed' if installed.
Richard
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUK4U33YTPMLOKBYKNTYMAPNJAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGUZTMOBTGQ>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
There was a problem in the 'utils/isam_build.py' where the definition of di_nrecords was long, which is 64 bits, CISAM expects an int (32 bits). I have pushed a change to fix the problem, alternately you can change the line 79 from: |
Hi Richard,
I pulled the repo and tried to build using the script, but I get a bunch of
error lines, originating with:
$ python3 utils/isam_build.py
/usr/bin/ld: cannot find -lifisam: No such file or directory
/usr/bin/ld: cannot find -lifisamx: No such file or directory
[...]
Is there any variable I should set or anything else to be done before
running the script?
BTW: from what I've seen you removed IBM isam libraries. Is it intended?
roberto
Il giorno sab 30 dic 2023 alle ore 19:01 Richard Moseley <
***@***.***> ha scritto:
… There was a problem in the 'utils/isam_build.py' where the definition of
di_nrecords was long, which is 64 bits, CISAM expects an int (32 bits). I
have pushed a change to fix the problem, alternately you can change the
line 79 from:
long di_nrecords
to:
{longsz} di_nrecords.
(Note the brackets are required, as the type of field is changed if your
running on a 32-bit architecture).
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUNCWY5X4H2GPRZZCO3YMBJGTAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGU3TKMBQGM>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Oops!! I managed to loose them, I have now re-added them, I left a .gitignore file that meant that they were not sent. Now fixed. |
Hi richard, no problem.
I could compile the library.
Now I get a different error:
$ python3 tables/lstisam.py
Traceback (most recent call last):
File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 37, in
<module>
lstisam('cgtotst1', 'tables')
File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 31, in
lstisam
ti, ii = fetch_isam_info(tabname, tabpath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/roberto/devel/python/pyisam/tables/lstisam.py", line 22, in
fetch_isam_info
idxinfo.append(tabobj.iskeyinfo(idxnum+1))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/roberto/devel/python/pyisam/pyisam/backend/cffi/__init__.py",
line 323, in iskeyinfo
self._chkerror(lib.iskeyinfo(self._isfd_, keyinfo, keynum+1), 'iskeyinfo')
File "/home/roberto/devel/python/pyisam/pyisam/backend/cffi/__init__.py",
line 199, in _chkerror
raise IsamFunctionFailed(ISAM_str(args), self.iserrno,
self.strerror(self.iserrno))
pyisam.error.IsamFunctionFailed: iskeyinfo: Illegal argument (102)
roberto
Il giorno sab 30 dic 2023 alle ore 19:45 Richard Moseley <
***@***.***> ha scritto:
… Oops!! I managed to loose them, I have now re-added them, I left a
.gitignore file that meant that they were not sent. Now fixed.
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUINZ2RJYWAOJZRAIWTYMBOOJAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGU4DEMZUGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Could you send me the tar file that you mentioned, as it has been removed from the comments, my email address is: I can then take a look. First impressions would imply that you have no keys defined on the table, that is zero. Thus asking for the first key will fail with the error that you gave. |
Hi Richard,
here you are!
Thank you.
roberto
Il giorno sab 30 dic 2023 alle ore 20:27 Richard Moseley <
***@***.***> ha scritto:
… Could you send me the tar file that you mentioned, as it has been removed
from the comments, my email address is:
***@***.***
I can then take a look.
First impressions would imply that you have no keys defined on the table,
that is zero. Thus asking for the first key will fail with the error that
you gave.
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXULWGVMVMT5FVZVP73TYMBTJBAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGU4DQNRZGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Your problem is that your adding one to the index number, then the pyisam also adds one to that. Thus when using the pyisam library, the first key is 0, which maps to key 1 to the ISAM iskeyinfo() function. I will send you the source code to a program which performs an also identical action as your lstisam.py. I will send you a tarball with my version of the script, which should be placed into the root directory of the environment, the script expects the data to be under exdata, but change that to suit. |
Thank you Richard.
Il Sab 30 Dic 2023, 21:44 Richard Moseley ***@***.***> ha
scritto:
… Your problem is that your adding one to the index number, then the pyisam
also adds one to that. Thus when using the pyisam library, the first key is
0, which maps to key 1 to the ISAM iskeyinfo() function. I will send you
the source code to a program which performs an also identical action as
your lstisam.py. I will send you a tarball with my version of the script,
which should be placed into the root directory of the environment, the
script expects the data to be under exdata, but change that to suit.
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUNT2GGB27CVXVMXOADYMB4MFAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGYYDGMJSGQ>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
I have now pushed my changes which will now pull a version of pyisam which can be used within a virtualenv using the script 'mkvenv' and support both CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared libraries in the repository, so that you no longer need to compile the extension modules before use. The README.md file has been updated with more useful information. According to the README that is part of the CISAM distribution, the use of isindexinfo() is deprecated due to the fact that the second argument is a void *, and there are now two separate functions, isdictinfo() which is the same as calling isindexinfo() with a key of zero, and iskeyinfo() which is the same as calling if isindexinfo() is called with a key of non-zero. I trust this will help you continue your work. |
Thank you Richard,
I am not at home right now. I will check it as soon as I get home.
Roberto
Il Mer 3 Gen 2024, 12:48 Richard Moseley ***@***.***> ha
scritto:
… I have now pushed my changes which will now pull a version of pyisam which
can be used within a virtualenv using the script 'mkvenv' and support both
CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared
libraries in the repository, so that you no longer need to compile the
extension modules before use. The README.md file has been updated with more
useful information. According to the README that is part of the CISAM
distribution, the use of isindexinfo() is deprecated due to the fact that
the second argument is a void *, and there are now two separate functions,
isdictinfo() which is the same as calling isindexinfo() with a key of zero,
and iskeyinfo() which is the same as calling if isindexinfo() is called
with a key of non-zero. I trust this will help you continue your work.
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUJ6AMHLD7I65J77XBDYMVARJAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVGI2DSNRYGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Hi Richard,
I am trying to read the records from my test table cgtotst1. As I do not
know ISAM very well I am trying to reverse engineer your python examples.
It would be wonderful if I had an example of creating a new table, reading
the records of an existing table and updating it (update, delete, add). I
will anyway try to do it by myself.
In the meanwhile I noticed one thing: it seems that vbisam does not report
the correct number of records.
I built the library twice, using ifisam and vbisam libraries. Here are the
results:
python3 tables/lstisam.py tables/cgtotst1
*Using cffi and ifisam library*
Library: IBM Informix C-ISAM Version 7.26.F
Num Keys : 3
Rec Size : 70
Idx Size : *1023*
Num Rows : *807*
$ python3 tables/lstisam.py tables/cgtotst1
*Using cffi and vbisam library*
Library: VBISAM 2.1.1
Num Keys : 3
Rec Size : 70
Idx Size : *1024*
Num Rows : *0*
As you can see, using the vbisam library 0 records are reported, while
using the ifisam library the correct number of 807 records is shown.
Also, the size of the index reported by vbisam is bigger then the one
reported by ifisam
roberto
Il giorno mer 3 gen 2024 alle ore 12:48 Richard Moseley <
***@***.***> ha scritto:
… I have now pushed my changes which will now pull a version of pyisam which
can be used within a virtualenv using the script 'mkvenv' and support both
CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared
libraries in the repository, so that you no longer need to compile the
extension modules before use. The README.md file has been updated with more
useful information. According to the README that is part of the CISAM
distribution, the use of isindexinfo() is deprecated due to the fact that
the second argument is a void *, and there are now two separate functions,
isdictinfo() which is the same as calling isindexinfo() with a key of zero,
and iskeyinfo() which is the same as calling if isindexinfo() is called
with a key of non-zero. I trust this will help you continue your work.
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUJ6AMHLD7I65J77XBDYMVARJAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVGI2DSNRYGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Hi, I can do that, if you could let me know how the .fld file stores the field information. I noticed that the first index on the example table has 0 key parts, this indicates that the table can be referenced using record numbers. The program 'isql' (which was a front-end for Informix, stored information about tables in tables named 'sys*', the field information was stored in 'syscolumns'. vbisam seems to always return zero rows when you request the dictinfo, seems that there may be a speed issue, as the original code called a function, tcountrows. Looking at the comments, it would appear that the tnrows is set to 0 or 1 depending on whether or not there are any records in the table, I assume that this is because, the program using the library is expected to perform the count itself. This will need some work to correct, as I have compiled the support in and now it gives a count of 65166 against 807 that CISAM gives, there must something else going there. As for an example, I assume that you want an example written in python, I will work on it and get back to you once finished. At the sametime I will look into what the isdictino() function of CISAM actually calls. Richard |
Please find attached a XZ compressed version of the CIMAM reference manual
original produced by Informix, it gives an overview of the various
functions within the library, but no the newer isdictinfo() and iskeyinfo()
functions which are simply the isindexinfo() function split into two, if
you passed a key number of 0 to isindexinfo() it calls isdictinfo(),
otherwise iskeyinfo(). The pyisam package provides convenience methods that
do this for you.
Richard
…On Thu, 4 Jan 2024 at 12:25, qippur ***@***.***> wrote:
Hi Richard,
I am trying to read the records from my test table cgtotst1. As I do not
know ISAM very well I am trying to reverse engineer your python examples.
It would be wonderful if I had an example of creating a new table, reading
the records of an existing table and updating it (update, delete, add). I
will anyway try to do it by myself.
In the meanwhile I noticed one thing: it seems that vbisam does not report
the correct number of records.
I built the library twice, using ifisam and vbisam libraries. Here are the
results:
python3 tables/lstisam.py tables/cgtotst1
*Using cffi and ifisam library*
Library: IBM Informix C-ISAM Version 7.26.F
Num Keys : 3
Rec Size : 70
Idx Size : *1023*
Num Rows : *807*
$ python3 tables/lstisam.py tables/cgtotst1
*Using cffi and vbisam library*
Library: VBISAM 2.1.1
Num Keys : 3
Rec Size : 70
Idx Size : *1024*
Num Rows : *0*
As you can see, using the vbisam library 0 records are reported, while
using the ifisam library the correct number of 807 records is shown.
Also, the size of the index reported by vbisam is bigger then the one
reported by ifisam
roberto
Il giorno mer 3 gen 2024 alle ore 12:48 Richard Moseley <
***@***.***> ha scritto:
> I have now pushed my changes which will now pull a version of pyisam
which
> can be used within a virtualenv using the script 'mkvenv' and support
both
> CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared
> libraries in the repository, so that you no longer need to compile the
> extension modules before use. The README.md file has been updated with
more
> useful information. According to the README that is part of the CISAM
> distribution, the use of isindexinfo() is deprecated due to the fact
that
> the second argument is a void *, and there are now two separate
functions,
> isdictinfo() which is the same as calling isindexinfo() with a key of
zero,
> and iskeyinfo() which is the same as calling if isindexinfo() is called
> with a key of non-zero. I trust this will help you continue your work.
>
> —
> Reply to this email directly, view it on GitHub
> <#1 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/ACXMXUJ6AMHLD7I65J77XBDYMVARJAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVGI2DSNRYGY>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB6UJZFNFOHM6QUKZXPX5SLYM2NTZAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGAYTIOBXG4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Hi Richard,
the .fld file is a proprietary file format from a niche and quite old
programming language. Its current name is verastream host integrator from
MicroFocus. Its original name is SuperNova. It is very powerful, and ahead
of its time, but as it comes with a license fee it is not very broadly used.
Therefore the format of the .fld is unknown to me, but I have received a
screenshot of the fields,
[image: immagine.png]
I would like to help my good friend to have python as an alternative to
that language.
All his data are in C-ISAM, which is blazing fast for his use case, faster
than any DBMS he tested.
roberto resecco
Il giorno ven 5 gen 2024 alle ore 13:56 Richard Moseley <
***@***.***> ha scritto:
… Hi, I can do that, if you could let me know how the .fld file stores the
field information.
I noticed that the first index on the example table has 0 key parts, this
indicates that the table can be referenced using record numbers. The
program 'isql' (which was a front-end for Informix, stored information
about tables in tables named 'sys*', the field information was stored in
'syscolumns'.
vbisam seems to always return zero rows when you request the dictinfo,
seems that there may be a speed issue, as the original code called a
function, tcountrows. Looking at the comments, it would appear that the
tnrows is set to 0 or 1 depending on whether or not there are any records
in the table, I assume that this is because, the program using the library
is expected to perform the count itself. This will need some work to
correct, as I have compiled the support in and now it gives a count of
65166 against 807 that CISAM gives, there must something else going there.
As for an example, I assume that you want an example written in python, I
will work on it and get back to you once finished. At the sametime I will
look into what the isdictino() function of CISAM actually calls.
Richard
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUMEFNJ5EPLC3ISAM4TYM72AXAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZYGYZDENRQGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Hi Richard,
I believe the attachment has been stripped from the email.
roberto resecco
Il giorno ven 5 gen 2024 alle ore 14:06 Richard Moseley <
***@***.***> ha scritto:
… Please find attached a XZ compressed version of the CIMAM reference manual
original produced by Informix, it gives an overview of the various
functions within the library, but no the newer isdictinfo() and
iskeyinfo()
functions which are simply the isindexinfo() function split into two, if
you passed a key number of 0 to isindexinfo() it calls isdictinfo(),
otherwise iskeyinfo(). The pyisam package provides convenience methods
that
do this for you.
Richard
On Thu, 4 Jan 2024 at 12:25, qippur ***@***.***> wrote:
> Hi Richard,
>
> I am trying to read the records from my test table cgtotst1. As I do not
> know ISAM very well I am trying to reverse engineer your python
examples.
> It would be wonderful if I had an example of creating a new table,
reading
> the records of an existing table and updating it (update, delete, add).
I
> will anyway try to do it by myself.
>
> In the meanwhile I noticed one thing: it seems that vbisam does not
report
> the correct number of records.
>
> I built the library twice, using ifisam and vbisam libraries. Here are
the
> results:
>
> python3 tables/lstisam.py tables/cgtotst1
> *Using cffi and ifisam library*
> Library: IBM Informix C-ISAM Version 7.26.F
> Num Keys : 3
> Rec Size : 70
> Idx Size : *1023*
> Num Rows : *807*
>
> $ python3 tables/lstisam.py tables/cgtotst1
> *Using cffi and vbisam library*
> Library: VBISAM 2.1.1
> Num Keys : 3
> Rec Size : 70
> Idx Size : *1024*
> Num Rows : *0*
>
> As you can see, using the vbisam library 0 records are reported, while
> using the ifisam library the correct number of 807 records is shown.
> Also, the size of the index reported by vbisam is bigger then the one
> reported by ifisam
>
> roberto
>
>
> Il giorno mer 3 gen 2024 alle ore 12:48 Richard Moseley <
> ***@***.***> ha scritto:
>
> > I have now pushed my changes which will now pull a version of pyisam
> which
> > can be used within a virtualenv using the script 'mkvenv' and support
> both
> > CFFI and CTYPES with either C-ISAM or VBISAM. I have kept the shared
> > libraries in the repository, so that you no longer need to compile the
> > extension modules before use. The README.md file has been updated with
> more
> > useful information. According to the README that is part of the CISAM
> > distribution, the use of isindexinfo() is deprecated due to the fact
> that
> > the second argument is a void *, and there are now two separate
> functions,
> > isdictinfo() which is the same as calling isindexinfo() with a key of
> zero,
> > and iskeyinfo() which is the same as calling if isindexinfo() is
called
> > with a key of non-zero. I trust this will help you continue your work.
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <#1 (comment)>,
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/ACXMXUJ6AMHLD7I65J77XBDYMVARJAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVGI2DSNRYGY>
>
> > .
> > You are receiving this because you authored the thread.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <#1 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AB6UJZFNFOHM6QUKZXPX5SLYM2NTZAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGAYTIOBXG4>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUJHIMAF5NNZTSHTCQLYM73DTAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZYGYZTGNJTG4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Roberto, Yes the image has been removed from the comments, could you send it directly to me at richard.moseley4@gmail.com, in the meantime, I've updated my additional isdictinfo() now uses a local copy of countrows() which now returns the correct row count, which I have now pushed. I am in the process of checking the difference between 32-bit and 64-bit version of the underlying ISAM formats. Richard |
I've now written a module called 'pyisam.tabdefns.fldfile' which will take one of the .fld files and generate the table definition from it, using a DynamicTableDefn. Since there is no field names or index information within the file, the module generates fields of the format 'fldNNNN' where the first field is numbered 1. The pyisam package will attempt to match the indexes when the first read is performed. I'm still working on a complete program that creates a new table, inserts data, updates data and deletes data, and finally removes the table again. |
Hi Richard,
did you reverse-engineer the .fld file? Wow!
roberto resecco
Il giorno gio 11 gen 2024 alle ore 17:08 Richard Moseley <
***@***.***> ha scritto:
… I've now written a module called 'pyisam.tabdefns.fldfile' which will take
one of the .fld files and generate the table definition from it, using a
DynamicTableDefn. Since there is no field names or index information within
the file, the module generates fields of the format 'fldNNNN' where the
first field is numbered 1. The pyisam package will attempt to match the
indexes when the first read is performed. I'm still working on a complete
program that creates a new table, inserts data, updates data and deletes
data, and finally removes the table again.
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUJK67TAM7KQFXUW2FLYOAFARAVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBXGQ4TCNBSHA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Yes, its quite simple, each element is 4 bytes stored LSB (for example 0x08 0x00 0x00 0x00), the first such element is the number of fields that follow, then for each field the first element is the type of data, and the the second seems to be screen size, which in combination with the image of the table you gave me, allowed me guess suitable types of field. If you look at _fldtype_map in the fldfile.py module this indicates that they are not all types are mapped. It is then a simple case of adding suitable TableColumn instances to the definition. There must be somewhere else that the field and index information is stored, as there is no field name or indexes within the .fld file. |
I do apologise for not getting back sooner, but as I was writing the missing support for inserting, deleting and updating records, I've encountered an issue over the creation of the underlying record buffer, which is present in the layer above the ISAMobject which is causing me a headache. |
Hi Richard,
No need to apologize. You are already doing so much.
Thank you
roberto resecco
Il giorno ven 9 feb 2024 alle 18:29 Richard Moseley <
***@***.***> ha scritto:
… I do apologise for not getting back sooner, but as I was writing the
missing support for inserting, deleting and updating records, I've
encountered an issue over the creation of the underlying record buffer,
which is present in the layer above the ISAMobject which is causing me a
headache.
—
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXMXUMGOTZVCQFMURYACFDYSZMG7AVCNFSM6AAAAABBGP7MO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZWGMYTINBZGE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
I have just made a push of code that ensures that the libvbisam library gets initialised correctly. Previously it was using data as left by the operating system, and happened to leave the file descriptor for the logging to zero, which when the first (or only) table was opened, the index file was opened and given the same file descriptor and hence all the logging information gets appended to the end of the index file incorrectly. The fix is to make the vbisam backend call the appropriate routine to establish the file descriptors as -1 to disable the logging by default. |
Hi, I know this repo is not updated since a lot, but I am trying anyway to ask.
I cloned the repo, compiled vbsam and tried to run some of the utils.
I am stuck at this:
Could you point me to what to do to fix this issue?
Thanks in advance
The text was updated successfully, but these errors were encountered: