New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CCKD64 support needed for cckddump/cckdload utils #167

Open
Fish-Git opened this Issue Dec 26, 2018 · 6 comments

Comments

Projects
None yet
3 participants
@Fish-Git
Copy link
Member

Fish-Git commented Dec 26, 2018

Hercules ships with two mainframe assembler utilities designed to allow users to dump an entire mainframe dasd volume to CCKD format for transfer to Hercules on their personal computer (cckddump.hla) and vice versa (cckdload.hla). Both are HLASM assembler source files living in the repository's "util" subdirectory. (*)

They were written by Greg Smith (the original author/designer of our CKD/CCKD and FBA/CFBA emulated dasd support) and were later updated by Chistopher Varlet in 2014 to support "very large dasds" (i.e. those with more than 64K tracks).

HOWEVER... Even though the cckddump/cckdload programs both now support volumes with more than 64K tracks thanks to Chistopher Varlet, they both currently only support the original 32-bit CCKD (Compressed CKD) file format.

This means if a "large" dasd volume containing more than approximately 75,000 tracks (i.e. more than 4GB of data when compressed, such as a "3390-9" or larger) cannot be compressed into a Hercules CCKD dasd image file of less than 4GB in size, then you're basically screwed. You can't transfer that volume from your mainframe over to Hercules on your personal computer because the size of the resulting CCKD file would be larger than the current 4GB CCKD limit. (The reverse is also true too: existing CCKD64 format dasd volumes on Hercules cannot currently be transferred to the mainframe either since the cckdload utility is currently designed to only supports the original 32-bit CCKD file format, not the new CCKD64 file format.)

Both of these utilities need to either be updated to support the new CCKD64 file format or else new CCKD64 versions of the utilities need to be created.


Note: This issue is a sub-issue (sub-project) of Issue #20: "We need 64-bit CCKD support"

@Fish-Git Fish-Git self-assigned this Dec 26, 2018

@Fish-Git Fish-Git changed the title CCKD64 support needed for cckddump.hla/cckdload.hla utils CCKD64 support needed for cckddump/cckdload utils Dec 26, 2018

@Fish-Git Fish-Git added Enhancement BUG and removed Missing labels Dec 26, 2018

@Fish-Git

This comment has been minimized.

Copy link
Member Author

Fish-Git commented Dec 27, 2018

HELP!

I'm stuck! :(

I've encountered an addressability error and I don't know how best to fix it.

In cckdlo64.hla (the CCKD64 version of cckdload.hla that I'm writing; see .zip file below for source and listing) there are two fields in a work area. One called l2tab and the other called tempbuf.

The l2tab field hold the CCKD Level 2 lookup table: a table of 256 Level 2 entries where each entry contains the dasd image file offset of the track data and the size and length of the track data.

The tempbuf field appears to be an I/O buffer or work buffer of some type where CCKD data is manipulated and/or accessed (read into and written from??).

The problem is, in the old non-CCKD64 version of the program (cckdload.hla), the start of the tempbuf was very close to the very end of available addressability to that work area, so even though the tempbuf field itself was 16384 bytes long, it was still addressable since it was the very last field in the work area.

The l2tab field, which is defined before the tempbuf field, under the old non-CCKD64 version of the program, was only 2048 bytes in size (256 entries of 8 bytes each: 4 for the dasd image file offset of the track data and 2 + 2 bytes for the size and length of that track data).

But with the new CCKD64 file format, each l2tab entry is now 16 bytes in size (i.e. twice as big): 8 bytes for the dasd image file offset of the track data, 2 + 2 bytes for the size and length of that track data, and 4 bytes of padding for alignment). That makes the size of the l2tab field 4096 bytes (256 * 16), thereby blowing addressability to the tempbuf field! (The tempbuf field is now unaddressable!)

No matter which field I place before the other, it makes the other one unaddressable! :(

I would like to move the 16K tempbuf field completely out of the existing work area dsect, and into its own STORAGE OBTAIN area, but I'm not experienced enough with z/OS programming to know how to do that, and I suspect doing so would only end up requiring another register to address that working storage anyway, thereby increasing the risk of a bug as a result. (I'm not familiar enough with the program's register usage to determine how much trouble (how risky) doing so would be.)

So I need some help. :(

Can any of you that are more experienced at z/OS programming help me out please? Can any of you figure out an elegant, low risk way of solving this problem?

Here's a zip file containing the new source code and the PDF listing that shows the problem. Page 64 of the listing is where the problem is.

I'll return to studying the code but I would really appreciate any help any of you could provide! :(

Thanks!

 
p.s. If you have a quality visual diff tool, you can compare my new cckdlo64.hla code against the existing cckdload.hla code and see that the changes that needed to be made (so far!(*)) were very minor. If you don't have a good visual diff tool, here's an ExamDiff Pro .htm report you can look at instead. 99% of the code is currently exactly identical.(**)

Thanks.


(*) I'm obviously not done making the needed changes yet! There are obviously more coding changes that need to be made. But what you're seeing is the first part of those changes. Once I get past this hurdle I can then start adjusting the code to correctly deal with the larger file offsets. But I need to get past this current problem first! Thanks.

(**) Ibid.

@Fish-Git Fish-Git added HELP! and removed IN PROGRESS... labels Dec 27, 2018

@mcisho

This comment has been minimized.

Copy link

mcisho commented Dec 27, 2018

I'll have a look. It might be a couple of days before you hear anything from me, I have several trips to airports to deliver/collect people coming up. If anyone else can offer a quicker turn-around, please feel free to do so.

@wably

This comment has been minimized.

Copy link
Member

wably commented Dec 27, 2018

@Fish-Git it looks like you basically have three choices.

  1. You could remove those large areas (L2TAB and TEMPBUF) out of the dsect completely; and obtain separate storage for those areas (via STORAGE OBTAIN), and then put word pointers in the dsect for those two areas. You would have to load a register with their addresses when you need them, of course. If you can't spare a register, then you could:

  2. Use the long displacement instructions. For example, using the code as currently assembled in the listing you provided in the .zip file above, you could address TEMPBUF this way (some examples):

     LAY     Rx,TEMPBUF
    

or,

     CLIY    TEMPBUF,C'X'

or,

     STY     Rx,TEMPBUF+4

even though it is beyond the normal base register range. There are a number of long displacement instructions - I believe that they all end in -Y for the mnemonic - that you can use to manipulate the contents of TEMPBUF in this way.

  1. If you could somehow spare a register, I would simply use a 2nd base register on the USING for this dsect and be done with it.

If I was coding this, I'd probably choose number 1 above. I can almost always find a register I can use temporarily without disrupting stuff, like R14 or R15, even if I have to load it a couple of times to restore its address after a macro invocation for example.

@Fish-Git

This comment has been minimized.

Copy link
Member Author

Fish-Git commented Dec 27, 2018

(Doh!)   %-P

I really should have taken the time to study the code a bit closer before panicking and crying for help. The solution was much easier than expected:

  1. If you could somehow spare a register, I would simply use a 2nd base register on the USING for this dsect and be done with it.

Yep!

Reviewing the code where the problem was occurring, I discovered there was indeed a register that wasn't being used! So I simply added a second register to the USING statement and inserted a pair of instructions to set it up. Problem solved!

Sorry to waste your time folks! I'll go crawl back under my rock now and leave you guys alone. :-/

(Sheesh, Fish! Can you possibly be more lame/incompetent?! Sigh)

@Fish-Git

This comment has been minimized.

Copy link
Member Author

Fish-Git commented Jan 1, 2019

I'm still working on this but it's taking me much longer than expected as the needed changes are more involved than I originally anticipated. (I'm having to change a lot of code to use 64-bit "grande" instructions in many places so it's going slow.)

I've temporarily removed "IN PROGRESS" status as I need to temporarily stop working on it to fix a Hercules QETH/OSA issue with my CTCI-WIN code that has cropped up. When I finish I'll return to plugging away on this issue again.

@Fish-Git Fish-Git removed their assignment Feb 16, 2019

@Fish-Git Fish-Git added the HELP! label Feb 16, 2019

@Fish-Git

This comment has been minimized.

Copy link
Member Author

Fish-Git commented Feb 16, 2019

I'm having real trouble finding the time to work on this issue, and would like to request help from somebody.

PLEASE!     (pretty please?)

It's bad enough that I am totally inexperienced with using the the new z/Architecture 64-bit instructions but I am also totally inexperienced with assembling, linking and testing these utilities on z/OS or similar guests.

It would be really great if someone else could take over this issue from me and finish it. I can send you what I've already got so far (which may or may not be right!).

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment