Bug Report: V3.1.4 on z/OS abends S0C4 for function fetch_both() and returns a python UTF decode error for function field_name() #852
Comments
pjfarleyiii
changed the title
pjfarleyiii
changed the title
|
Has anyone had the time yet to review these errors enough to know if they are environmental to the Zxplore system or actual ibmdb issues? Would appreciate knowing if there is any further information I could provide to assist with analysis and remediation. |
|
@pjfarleyiii I tried on my local machine and i didn't face any problem. Looks like issue is because of encoding. Can you please check what is the encoding you have. |
|
I am not sure which encoding the DB2 subsystem is using, but the general encoding I use on the Zxplore z/OS system is "IBM-1047" or in python "cp1047". Both of the test scripts attached to this report are stored in the Zxplore USS file system as IBM-1047. If you can tell me how to query it I will try to find out what the Zxplore DB2 subsystem is using by default, but I am kind of assuming it is probably code page 037. |
|
I found out how to display the CCSID's in use by DB2 at the Zxplore site. SPUFI output from that site follows. |
|
@pjfarleyiii, I hope now are you able to run your test scripts correctly. Please let us know if you need any further help. Thanks |
|
@Earammak I do know about file tagging and my scripts are tagged IBM1047. They fail as coded and tagged and described in detail in my first post. What can I do to help your team to fix these errors? The fetch_both script fails at the fetch_both function and the fetch_name script fails at the fetch_name function, and there is no encoding used to call those functions. |
|
@pjfarleyiii, I'm able to run in my environment. I used the same python script as your shared. Please let me know if there is any discrepancy in my script. Thanks |
|
@Earammak the script outputs you got are not correct, even though they did not get any error. The "fetch_name" script is supposed to print the number of columns returned for each row and the names returned by the "fetch_name()" function for each column of each row. Your execution printed nothing, as if your DB2 data table is empty. Please try it with a DB2 table that contains some data and see what you get. The same is true for the "fetch_both" script, if the DB2 table is not empty it should print the row number of each row returned by the "fetch_both()" function. Again, please try that script with a DB2 table that has some data in it and see what you get. Here are modified copies of the test scripts that use table SYSIBM.SYSCOLUMNS, which should generate LOTS of output if the fetch_both() and fetch_name() functions are operating correctly. On the Zxplore system there are over 6400 rows in table SYSCOLUMNS, your system may have more or less rows. I found a set of WHERE conditions that will reduce the output to a reasonable size, as follows:
db_fetch_tuple_no-err-2.py.txt Please try these scripts on your system and let me know what you see. The new script db_fetch_tuple_no-err-2.py gets no error on the Zxplore system and prints 42 rows of output. It only uses the P.S. - The modified uploaded scripts above are encoded as ISO8859-1, not IBM1047. The originals on the USS file system are encoded as IBM1047. |
|
@pjfarleyiii, Find the below logs for the above-mentioned scripts. I'm able to run them successfully, and don't see any problem. Thanks |
|
I do see that your system runs the scripts without any errors, and the output is correct for what is coded. See my original report at the top. The "fetch_both" script abends at the first call to the "fetch_both()" function with S0C4 (addressing exception) in something called "TOROLABA" somewhere deep inside of the python runtime code. The "fetch_name" script ends with a decoding error at the first call to the "fetch_name()" function. I have generated the error output pasted below using both of the modified scripts. These errors are consistent and repeatable - I get an S0C4 abend any time I use the "fetch_both()" function and a decoding error any time I use the "fetch_name()" function, no matter which DB2 table is referenced in the SELECT statement. This is all happening on the IBM Zxplore z/OS system, not on a personal z/OS system of my own. I would appreciate any pointers you can provide for me to get more information than the dump I provided in my original report to help you to resolve these errors, or what I need to ask the Zxplore administrators to do that would help you to resolve them. |
|
@pjfarleyiii Since you are observing this problem only on IBM Zxplore z/OS system and it is working on your local system. Thanks |
|
@Earammak You have it all wrong. I have NO LOCAL SYSTEM to run these scripts on I ONLY HAVE ZXPLORE!!! WHAT CAN I DO TO GET MORE INFORMATION FOR YOU TO BE ABLE TO SOLVE THE ISSUE I SEE ON ZXPLORE? The Zxplore admins have said they will cooperate to help me solve the problem, but I have to know what to ask them to do or what I have to do to give you the information you need to solve the problem. I sent you a dump, is that not enough information? What more do you need to solve the issue? Please help me solve this issue. [Edit]: Do the Zxplore admins have to open a support ticket with IBM or with you to get help to solve the problem? |
|
Was a SVC dump taken? If yes, can we get that, please? If no, can you set a SLIP for the S0C4 and capture a dump for us, please? |
|
I cannot do that (I do not have the privileges needed), but I can ask the Zxplore sysadmin to set a SLIP and capture the dump for you. I can create the S0C4 on demand, so it should not take too long to get it. I will post it in a zip file when I have received it from the Zxplore team. |
|
Thanks! Do you know if the Zxplore team have plans to upgrade Python, as v3.9.2 is ~2 years old? |
|
As I understand it from the Zxplore admins, it isn't entirely up to them - they use some "common" base system generated by some other IBM group (possibly WAZI?) and have to wait for that "common" base to be updated, they aren't always allowed to do upgrades on their own. Meanwhile I am still waiting on a response to my request to set up a SLIP trap. As this python package isn't on their "we will positively support this" list, it may take a bit longer to get the dump. They did say they would try to help, but we have to depend on their availability to coordinate with me to get the dump, and there are time zone issues. I think my main contact is on Zulu time (UTC 0 offset) and I am UTC -4 (EDT). |
|
They are likely using the IBM ADCD distribution also used by other IBM solutions. If you can find HLQ ADCD in ISPF 3.4 you can determine the ADCD build from the second LQ. For example, the existence of ADCD.Z25D suggests it is the May 2023 distribution mentioned above. Can you also confirm you're running on real hardware? I see machine type 3931 from uname, so likely a z16? The ADCD distribution is often used by emulated environments which have machine types 1090-1092. The reason why I am asking is that I had another customer report a similar issue a while ago. They opened a support case with IBM Open Enterprise Python support, but the issue was never diagnosed as a refresh of their system to latest maintenance resolved the issue. |
|
Yes, I believe they are running on a real z16, there was a hardware upgrade last month or the month before where they said they were going to a z16. No datasets under HLQ ADCD though. By the way, would a SYSMDUMP file be as helpful as an SVC dump? I believe I can replicate the problem running the python3 script under BPXBATSL, and I can run that and generate the SYSMDUMP myself without any help form the admins. [Edit] I tried to set up a job to capture a SYSMDUMP, but it does not do that, only generates CEEDUMP in my $HOME directory even when I override LE option TERMTHDACT to UAIMM. It does not work at all using BPXBATSL (/bin/login fails then) and BPXBATCH runs the shell in another address space. Will have to wait for the SLIP to get set up by the admins. |
|
Progress report: The Zxplore admins set up a SLIP as requested and I generated the S0C4 twice, but the SLIP did not fire yet. I will continue to work with the Zxplore admins to get the dump generated from the SLIP and post here again when we have it. |
|
Progress report: Unfortunately no more progress as yet. Still trying to get the repeatable S0C4 to trigger the SLIP. Administrators have been busy with other Zxplore issues, but I hope to have something available soon. |
|
Maybe its a similar issue, but any ideas why running on z/OS (printing ibm_db.conn_errormsg() ) would create a conversion error: And given the error reported above also occurs at conversion, is there a chance that something returned by the odbc driver is EBCDIC and as such not converted correctly? 0xe4 is 'ä' but I recall from using 1047 codepage at 3270 terminals that could be a bracket such as [] or {} displayed as 'ä'. It could be environmental, wouldn't be the first time that anything else than cp037 would not work. Any environment variable or odbc setting I can look at or try to get rid of the conversion error? It would be awkward, if one had to do ibm_db.conn_errormsg().decode(???) to get rid of the error. |
|
@denisgaebler I had that same problem when first trying to issue an "exec_immediate()" to DB2. Below is what I had to code to see the error message from the failed operation. Part of the error message seems to be encoded in UTF-8, part seems to be encoded in IBM-1047. Perhaps something similar will work for a connect error for you. |
|
@pjfarleyiii previously opened a ticket for |
|
@daveyc, As I responded to you on IBM-MAIN, IMHO it is not a useless exercise to nail down the root cause of these errors. If that root cause turns out to be a broken python version on the Zxplore system, that provides us with proof to goad the Zxplore admins to make a request that their version of python be upgraded. As far as I am aware, they do not actually completely control the contents and versions of products on the system they are using to reach out to those who wish to learn more about the z/OS environment and quirks. Their mandate is to teach and help people learn. My quest here is to make a useful product (ibmdb) work in that teaching environment for my own learning benefit and the benefit to other students seeking to learn there. |
|
@pjfarleyiii I think we should close this issue. Firstly, and most importantly, Python 3.9.2 is no longer available so we can't test it at Rocket. I'm also dubious about the Python distribution on the Z XPlore system. The traceback contains path names such as
100%. But there is nothing the maintainers of this repo can do for an unsupported and broken Python. |
|
@pjfarleyiii Can you please provide update on this? |
|
@daveyc Another user hit the similar problem. Detail is as below: They have shared the SVCDUMP files that you can access from ftp2.rocketsoftware.com by searching TS013773510. Please download the dump and have a look into it. Thanks. |
|
@Earammak , my status is that I am still awaiting a response from the Zxplore admins about whether the SLIP and the SVCDUMP DASD space are available and active. I have also tried generating the dump using
but I have yet to see an SVCDUMP get created from the abend process. I am afraid this issue is just not very high on the list of things the Zxplore admins are concerned with. If the other dump that was submitted is not sufficient to resolve the issue I can continue to try to get an SVCDUMP from Zxplore. Please keep me posted on whether or not I should continue to try to get this dump. |
|
Can you add the following to the odbc.ini file: If you're not authorized to update /etc/odbc.ini, you can copy the file to your local directory and make the changes there. Make sure to change the environment variable DSNAOINI accordingly. Then rerun the python program. This will produce a file named "DD:DSNAOTRC" in the current directory (yes, it has a weird name). Then format the trace as follows: and upload the trace.txt file here. |
|
Please also post the output of: |
|
@jthyssenrocket , here are the requested "set | grep" results, working on the rerun with diagnostics next. |
|
@jthyssenrocket the DIAG entries you requested were put into private copy of the /etc/odbc.ini file in the $HOME/.db2/odbc.ini: No file named "DD:DSNAOTRC" was created, but I did get a SYMDUMP-format CEEDUMP in the MVS dataset area named "Z99999.CEEDUMP.D221.T1030416.P00103DC" due to the _CEE_RUNOPTS options @daveyc suggested. There is no executable named DSNAO64T in this system's DSNC10.SDSNLOAD, but there is a module named DSNOATRC. Should I look up how to run DSNOATRC in JCL and feed it the CEEDUMP as input? |
|
Sorry, my mistake: DSNAO64T is in SDSNLOD2 (not SDSNLOAD). However, we need the DD:DSNAOTRC file first. There is an alternative set of instructions here: If it doesn't work for you, I'll need some help from IBM Db2 ODBC dev to get the ODBC diag trace captured, as it contains information about the options used including application encoding scheme. |
|
The alternative instructions actually works for me from TSO OMVS, but not from a putty ssh session. |
|
I will try the alternative instructions from a USS login and reply back. I am assuming that DSNAO64T is what I should use for this test, correct? I created a Rexx script to allocate files including DSNAOTRC and a shell script to run the "dsn" commands and the python script. I allocated a userid.DIAGTRC file under my TSO ID defined as FB80 as documented in that IBM doc link you posted. Zxplore SMS rules for "students" on the site prevent us from creating or writing large sequential files though, so the allocated file may not be large enough to hold a lot of data. I can inquire of the admins if they can permit me an SMS exception that would allow a larger file to be created for the DIAGTRC output. A file named "DD:DSNAOTRC" does seem to be created in my USS $HOME directory, but it is zero bytes long. The userid.DIAGTRC file I allocated is also empty after all of my attempts. I also tried the shell script from OMVS as well, but same results as other tries. Output from OMVS attempt pasted below. Following are the EXEC and shell script contents and the STDERR output from running them from TSO Option 6 as "EXEC (ODBCTEST) EXEC". I even tried running BPXBATCH of the shell script in a batch job with REGION=500M, but the results were the same, "Not enough memory" error in STDERR. userid.EXEC(ODBCTEST): /z/z99999/odbctest.sh: /z/z99999/odbc.stderr: OMVS Output: |
|
Sorry, if it wasn't clear: there are two alternative ways to collect the trace Option 1: add the DIAG* keywords and optionally allocate DD DSNAOTRC file. It should be written during processing. Option 2: remove the DIAG* keywords, and use dsnao64t: The error message "Not enough memory to hold the trace." suggests that your TSO region size (or imposed OMVS limits) is too small. |
|
OK, I tried option 2 from a USS ssh login, results pasted below. I will try from an OMVS shell in TSO but I suspect the REGION for TSO is smaller than the USS limit on this "student" system. Same results for both options from an OMVS shell in TSO logged on with |
|
On my system Option 1 works from both TSO OMVS and ssh into USS. On my system Option 2 only works from TSO OMVS. I do have a TSO REGION size of 400.000. Do you think it would be possible to get a larger TSO region size? Other potential alternative: in a previous post #852 (comment) you wrote that you did not get an abend with fetch_tuple. Edit: fix typo in SQL statement |
|
I did do a TSO OMVS using Alternate SQL SELECT run using fetch_tuple, and succeeded after changing your suggestion to |
|
1047 is EBCDIC (US codepage), so the python program is still using EBCDIC despite setting CURRENTAPPENSCH=UNICODE in the ODBC ini file, which is what I suspected. This is why the ODBC diag trace is critical, so we can see why ODBC decides to use EBCDIC instead of the specified UNICODE. Would it be possible to retry "option 2" with your new larger TSO REGION size? FWIW, On my system the same program returns: With CURRENTAPPENSCH=ASCII I get: |
|
Done when I got back home today, unfortunately none of them seems to work. Option 2 results in TSO OMVS session using Option 2 results in TSO OMVS session using Option 2 results in TSO OMVS session using |
|
What is the tagging of your ODBC ini file? ls -lTE $DSNAOINI |
|
I suggest we update https://github.com/ibmdb/python-ibmdb/blob/master/install.md section 2 as follows:
Example: I assume the same is true for node.js ibm_db on z/OS, so I suggest adding
|
|
@jthyssenrocket , Current encoding of the Zxplore global /etc/odbc.ini file is pasted below. THIS IS THE SOLUTION!!! I updated my private copy of the odbc.ini file to remove the text attribute and the script that abended S0C4 using the [Edit] Option 2 of the "get trace information" procedure discussed earlier in the thread now also works correctly in an ssh session to USS. I successfully created a "trace.txt" file using that procedure (now that we do not need it, of course). I will ask the Zxplore admins to change the text attribute of the global $DSNAOINI file with the command Out of pure curiosity however, what is the technical reason that the odbc.ini file cannot have the text attribute? Thank you all very much for your patience with me and with the response time from the Zxplore admins. As far as I am concerned this problem is solved and the issue can be closed, but I leave it to the committers here to close the issue after the recommended documentation changes are done. I will continue to test ibmdb functions on the Zxplore system and report separately any other issues that I find. Best regards, Peter J. Farley III Zxplore global $DSNAOINI file characteristics as of 08/14/2023: |
|
Annoyingly, I've had the note about the ODBC ini file in my Python ibmdb on z/OS presentation for several years, but this is the first time I've seen the issue materialize as an S0C4. Unfortunately it appears there is no easy way to programmatically check the application encoding scheme after connecting to Db2, but before issuing the first SQL statement. |
|
@pjfarleyiii can you please prod the ZXplore admins to upgrade Python? A browse through to Python SDK fixlist flags multiple fixes for encoding issues. You may find that other committers are not so accommodating to help debug issues like this.
I suspect it opens the file in binary mode. I tag my |
|
@daveyc, I have already asked about upgrading the python installation on Zxplore. The answer I received is that the Zxplore z/OS image is derived from another IBM team's "base" image, and the version of python which is available on Zxplore is installed by that other team in their image. When that other team's "base" image is updated and merged into the Zxplore system is when the version of python on Zxplore will be updated. You also need to understand that none of the "students" on the Zxplore system (and I am one of those) has permissions or disk space quotas that allow installing even other python packages (among other things, venv is not available to try to do that "on your own"). Thus I am unlikely to be in a position to find issues in other python packages, at least not on Zxplore. In fact, the existence of the ibmdb package and your pyzfile package on Zxplore are due to the admin there accommodating my request to install them at the global level when he had some time available. |
I understand your perspective. It's unfortunate that your workplace doesn't provide a more advanced USS development environment, considering you're challenging the notion that experienced individuals can't pick up new skills. |
|
I don't have an /etc/odbc.ini file on my system, so it must be something the Zxplore (or the owners of the installation they inherited) has created. I can't find any sample odbc.ini file in /samples or in /usr/lpp/db2 either. Would it be worthwhile to ship a sample ODBC ini file with ibm_db that gets installed into site_packages with the proper tagging? There is an Aha idea open to have ibm_db included in the IBM python distribution (see https://ibm-z-software-portal.ideas.ibm.com/ideas/PY-I-7). This would also be an opportunity to ship a sample ODBC ini file with proper tagging. |
|
Interestingly the choice of shell also impacts which encodings can be used...
|
|
@jthyssenrocket, isn't the presence (or absence) of the odbc.ini file a result of the z/OS installation installing (or not installing) the ODBC driver for DB2? Isn't that part of the DB2/ODBC installation process? I have never installed DB2 or an ODBC driver so I am clueless on the procedure myself. Certainly if ibmdb gets shipped with python itself that would be an ideal place to populate a sample odbc.ini file. Does the difference in the behavior under different shells change for |
|
@daveyc, yes, I am bummed that my employer doesn't even have python installed, much less a USS development environment. Some things we just have to "work around" on our own to keep on learning. Continuously learning new things is what keeps old-timers like me alive and kickin'. I've always said that if I enjoy what I do 50% of the time, I am better off than 80% of working stiffs, and the other 50% is just what you have to put up with in any employment position. Learning new stuff is well up there in the 50% I enjoy. |
|
Not sure, but I think you should be able to reproduce the table above if you start "sh". I rarely use IBM /bin/sh as it is too painful :) I am not able to find odbc.ini on my system (neither in /etc, /samples or /usr/lpp/db2) |
|
@jthyssenrocket, 1000% agreed, IBM /bin/sh is WAY too painful. If I get some time I will try to see if |
|
Status report from Zxplore: Their Again, my thanks to everyone here for your patience and help. |
The errors described in this issue are occurring on the IBM Zxplore open learning system for z/OS (anyone can register to join at Zxplore Learning System). I am not an admin of that system and have no access to any software on that system except what the IBM Zxplore admins provide. I have no authority to install or upgrade software on that system, only the IBM Zxplore admins have that authority.
Please advise what I can communicate to the Zxplore admins to get assistance in resolving these issues.
Operating System Name: z/OS V2.4
db2level output from clidriver if in use:
Target Db2 Server Version: V12
Python Version: Python 3.9.2
ibm_db version: 3.1.4
For non-Windows, output of below commands:
uname OS/390
uname -m 3931
Value of below environment variables if set:
IBM_DB_HOME: DSNC10
PATH: /usr/lpp/IBM/cvg/go/bin:/usr/lpp/IBM/cyp/v3r9/pyz/bin:/usr/lpp/IBM/zoautil/bin:/usr/lpp/db2c10/jdbc/bin:/bin:/z/bin:.:/usr/lpp/java/J8.0_64/bin:/usr/lpp/zLiberty/V8R5/wlp/bin:/z/z02446/bin:/z/rocket/tools/bin:/usr/lpp/ported/bin:/usr/lpp/IBM/cnj/v18r0/IBM/node-latest-os390-s390x/bin:/usr/lpp/IBM/cnj/v18r0/njsc/bin:/global/node/bin:/usr/lpp/mqm/V9R1M1/web/bin:/usr/lpp/IBM/cobol/igyv6r4/bin
LIB/LD_LIBRARY_PATH/DYLD_LIBRARY_PATH: /usr/lpp/db2c10/jdbc/lib
Test script to reproduce the problem.
For fetch_both() attached script db_fetch_both_err.py as text file
For field_name() attached script db_fetch_name_err.py as text file
db_fetch_both_err.py.txt
db_fetch_name_err.py.txt
Steps to Reproduce:
Output from test scripts for functions fetch_both() and field_name() (Zxplore userid masked) pasted below. Also uploaded here is the CEEDUMP output from the S0C4 abend as a zip fille.
CEEDUMP.20230528.175140.16842984.zip
The text was updated successfully, but these errors were encountered: