Skip to content

v1.1.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 14 Sep 10:33
· 356 commits to main since this release
v1.1.0
23011cd

FTPD 1.1.0 is about uploads: how much space they get, what happens when they run out of it, and where the data set lands. Eleven issues closed — #100, #121, #122, #124, #127, #128, #129, #133, #135, #143 and #148 — three of them reported from outside, and one change that touches every installation. It is not a drop-in for 1.0.x: the data set names lose their version qualifier, so the started task's STEPLIB and the APF entry are re-pointed once. Your FTPDPRM0 needs no change — no keyword changed meaning, and the three new ones default to the old behaviour. Built against libc370 1.0.6 and libufs from ufsd 1.3.0 (the build requires ufsd >= 1.3.0; UFSD itself stays optional at runtime, see the guide's prerequisites). The FMID is TFTP110 and it deletes TFTP100.

This release was published once and withdrawn. If you installed the
earlier build, or tried to, read Upgrading from 1.0.x below — the procedure
has changed and is now shorter, not longer.

Read this first: what an operator or a client notices

  • The data sets are FTPD.LINKLIB, FTPD.SAMPLIB and FTPD.AFTPDLOD now (#121), not FTPD.V1R0Mx.*. After the install, scratch the old libraries, re-point STEPLIB and update the APF entry — the install job cannot do that for you, and until STEPLIB moves you are still running 1.0.x with no error anywhere.
  • No migration step before the install (#143). The SYSMOD carries ++VER(Z038) DELETE(TFTP100); SMP transfers ownership of the module itself during the APPLY.
  • On an upgrade the APPLY step ends COND CODE 0004 on HMA2461 SYSMOD TFTP100 NOT FOUND ON SMPSCDS LIBRARY. Expected; the install job gates its ACCEPT accordingly. A fresh install ends RC 00.
  • An upload that runs out of space answers 552, not 451 (#129, #135). 552 is permanent: a conforming client no longer retries it. A genuine write error still answers 451.
  • A failed upload no longer leaves its data set behind (#129). A data set FTPD allocated for the transfer is scratched (FTPD074I); an upload into an existing data set is never scratched.
  • Unused space is released at close (#128) for data sets FTPD allocated itself.
  • DEFUNIT and DEFVOLUME are honoured now (#133), and so are SITE UNIT= and SITE VOLUME=. If your parmlib names a volume, new data sets land there from this release on; a volume that is not online is reported at startup (FTPD059W) and ignored.
  • STOR space is configurable (#100): DEFPRIMARY, DEFSECONDARY and DEFSPACETYPE in FTPDPRM0, SITE PRIMARY=/SECONDARY=/TRACKS/CYLINDERS/BLOCKS per session. The defaults are the old hardcoded TRK(10,5).
  • A UFS upload that failed no longer answers 226 (#148): 552 for out of space, the UFS error for any other write failure, 426 for a broken data connection. Expect the 552 at about 4.06 MB per file — a property of the UFS on-disk format, not a full disk.
  • An IEAAPF00 entry for FTPD.LINKLIB dictates its volume. If you already have one, put VOL=SER= on the allocation job before you run it (see Install).

Fixed, and measured on MVS

before now
an upload past the allocation (#127, #129, #135) SB37/SD37/SE37, answered 451 — retryable, and every retry met the same full data set 552, permanent, with advice on how to allocate more
STOR space (#100) SPACE=TRK(10,5), fixed — about 85 tracks, ~4 MB at FB/80/3120 DEFPRIMARY/DEFSECONDARY/DEFSPACETYPE, overridable by SITE
an unparseable space value silently 10 in one code path, 100 in another refused, with a message naming the default it fell back to
the session after an SD37 (#129) 300 seconds to a client timeout; the DD left allocated, so the scratch got IDCAMS RC=8 ends promptly; the data set is gone; a retry from a fresh allocation works
the data set of a failed upload (#129) left catalogued, holding the partial upload scratched, FTPD074I
unused space after a small upload (#128) kept released at close
TYPE A upload, more than 256 bytes per receive (#122, #124) lines missing, fragments spliced together 5 rounds of 2 MB: byte-identical every time
DEFUNIT, DEFVOLUME, SITE UNIT=, SITE VOLUME= (#133) parsed and ignored honoured; configured values checked at startup
SITE VOLUME= naming a volume that is not mounted (#133) session hung, IEF238D outstanding on the console a reply to the client
a failed UFS upload (#148) 226 Transfer complete 552, the UFS error, or 426
data set names (#121) FTPD.V1R0M2.LINKLIB, … — new names every patch FTPD.LINKLIB, … — one APF entry, one IPL, ever
replacing the previous FMID (#143) the withdrawn first build: a manual step before the install, silent when skipped DELETE(TFTP100) during the APPLY
1.0.x 1.1.0
FMID TFTP100 TFTP110
Libraries FTPD.V1R0M2.LINKLIB, … FTPD.LINKLIB, …
libc370 1.0.4 1.0.6
libufs (ufsd) 1.2.2 1.3.0

Install

Download ftpd-1.1.0-dist.zip (or .tar.gz); the guide and every file you need are inside:
README.md the installation guide
ftpd-1.1.0-load.xmit the FTPD load module
ftpd-1.1.0-samplib.xmit the STC procedure and the configuration pattern
ftpd-1.1.0-alloc.jcl allocate the data sets — run once
ftpd-1.1.0-inst.jcl receive, apply, accept

Upload the two XMITs in binary, submit the allocation job once, then the install job — the only thing to edit is the two data set names you uploaded to. The SYSMOD travels inline in the job, so there is no third file to transfer. FMID is TFTP110.

The documentation in the repository:

One thing to check before you run the allocation job. On MVS 3.8j an IEAAPF00 entry is dsname volser, not a name alone. If you have already added FTPD.LINKLIB to the APF list, that entry does not merely permit a volume — it dictates one, and the allocation job says UNIT=SYSDA with no VOL=SER. Put the volume in:

//LINKLIB  DD  DSN=FTPD.LINKLIB,DISP=(NEW,CATLG,DELETE),
//            UNIT=SYSDA,VOL=SER=your-apf-volume,

Get it wrong and you have a library that exists, holds the right module, passes every check — and is not authorised. Nothing reports it, because FTPD falls back to the SVC 244 route and starts anyway. The startup message is the only place the two are distinguishable: on a system where you added an APF entry, FTPD008I AUTHORIZED BY SVC (MODULE KEY 8) means the entry is not covering the library. Found the hard way during an install of UFSD 1.3.0 on a development system.

The two caveats from 1.0.x still apply to the shipped material:

  • On TK5, SYS2.PARMLIB does not exist — put FTPDPRM0 in SYS1.PARMLIB and start with /S FTPD,D='SYS1.PARMLIB'.
  • The two JES DDs in the procedure, HASPCKPT and HASPACE1, carry the unit and volume of the system FTPD was built on. Point them at your own JES2 checkpoint and spool, or delete both if you do not want the JES interface. Both are required together: with HASPACE1 missing, every JES command answers 451.

Not using SMP? Receive ftpd-1.1.0-load.xmit into a library and point the procedure's STEPLIB at it.

Upgrading from 1.0.x

Install it. The FMID takes care of itself — that is #143, described below.

What the install job cannot do for you is the data sets, and the reason is worth thirty seconds. SMP records the ddname an element was installed through, never the data set name. Both FTPD.V1R0M2.LINKLIB and FTPD.LINKLIB end in LINKLIB, so both installs use ddname LINKLIB — and when 1.1.0's APPLY deletes the old load module it resolves that ddname in its own job, which points at the new library. FTPD.V1R0M2.LINKLIB is never opened.

So after the install:

  1. Scratch the old libraries. Look them up rather than assuming the patch level — ISPF 3.4 on FTPD.*, or LISTCAT LEVEL(FTPD). Expect three: FTPD.V1R0M?.LINKLIB, .AFTPDLOD, .SAMPLIB. They are invisible to SMP now, so nothing will ever mention them again.
  2. Repoint the started task's STEPLIB at FTPD.LINKLIB. This is the step that decides whether you are running 1.1.0 at all. If you skipped step 1 the old library still exists and still holds a working module: FTPD starts, serves, and logs no error — it is simply the old one.
  3. Update the APF entry. The last time it will need changing, because the name no longer carries a version. One IPL ever, not one per release.

Stop FTPD before you begin: FTPD.LINKLIB is the library it loads from.

One return code that looks wrong and is not. On an upgrade the APPLY step ends COND CODE 0004, not 0000:

HMA2270 APPLY PROCESSING SUCCESSFULLY COMPLETED FOR SYSMOD TFTP110
HMA2461 SYSMOD TFTP100 NOT FOUND ON SMPSCDS LIBRARY
HMA2050 APPLY PROCESSING COMPLETED - HIGHEST RETURN CODE IS 04

HMA2461 is SMP noting that the deleted level has no backup entry to keep — it never will, because it was deleted rather than superseded. HMA2270 is the line that says the APPLY worked. The install job expects this and gates its ACCEPT accordingly. A fresh install has nothing to delete and ends RC 00.

Your FTPDPRM0 and your RAKF definitions are unaffected. No keyword changed meaning; the three new ones default to the old behaviour.

What is running

FTPD005I stamps the library the module was actually linked against, so a deploy that did not take cannot hide:

FTPD000I FTPD 1.1.0 (23011CD) STARTING
FTPD005I LIBC370 1.0.6

If that second line still says 1.0.4, the STC is running the 1.0.2 module.

Full changelog

Why the first build was withdrawn (#141, #143)

The withdrawn build carried a migration step that had to be run by hand before the install job, and warned about it at length: SMP keys element ownership on the module name, so a new FMID could not replace an element the old one owned. Skip the step and RECEIVE, APPLY and ACCEPT all reported success over an empty target library.

meltdown03 hit it within the day:

I get an empty FTPD.LINKLIB dataset only after running the FTPDALC and
FTPDINS jobs completed with CC 0000.

followed by "looks like I did not read the instructions and need to do extra steps" — which is the correct diagnosis and exactly the wrong shape for an install job. A step that is mandatory, silent when skipped, and documented somewhere else is a step that will be skipped.

It is gone. The SYSMOD now carries ++VER(Z038) DELETE(TFTP100) and SMP transfers the ownership itself during the APPLY. Nothing has to be prepared and nothing has to be removed first. Measured on a development system both ways, with throwaway ids and throwaway module names, in the shape this release actually has — same ddnames, different data sets:

HMA2240 SUCCESSFULLY DELETED LMOD FTPD ON LINKLIB LIBRARY
HMA2380 COPY SUCCESSFUL - MOD=FTPD - LMOD=FTPD - LIBRARY=LINKLIB - ...
HMA2270 APPLY PROCESSING SUCCESSFULLY COMPLETED FOR SYSMOD TFTP110

That came with a policy change: one FMID per release, each deleting its predecessor. TFTP110 is 1.1.0 and nothing else. A SYSMOD id is seven characters, three of which are the version digits, so no version component can ever exceed 9.

Added

  • Configurable STOR space (#100). Three new FTPDPRM0 keywords, named after the z/OS FTP.DATA statements they mirror:

    DEFPRIMARY=10          1-16777215
DEFSECONDARY=5         0-16777215; 0 means no secondary extents
DEFSPACETYPE=TRACK     TRACK | CYLINDER | BLOCK  (TRK/CYL/BLK too)

    The defaults are deliberately the old hardcoded numbers, so an existing installation that adds nothing to its parmlib behaves exactly as before. A system that receives large files wants something like DEFSPACETYPE=CYLINDER, DEFPRIMARY=10, DEFSECONDARY=10 — 160 cylinders, about 2400 tracks. Note that the 16-extent limit (1 primary + 15 secondary) and the single volume stay either way, so a larger PRIMARY buys more than a larger SECONDARY. Each is validated at startup and reported in the CONFIG dump:

    FTPD058I   DEFPRIMARY=10 DEFSECONDARY=5 DEFSPACETYPE=TRACK

    SITE PRIMARY=, SITE SECONDARY= and SITE TRACKS/CYLINDERS/BLOCKS override them for the rest of the session, and a bare keyword resets to what the parmlib configured rather than to a built-in constant. The fallbacks that used to paper over an unparseable value are gone: a bad number is refused with a message naming the default it fell back to, instead of silently becoming 10 in one code path and 100 in another — both of which were really in 1.0.2.

    Keyword Effect
    SITE PRIMARY=n primary space for the next data set
    SITE SECONDARY=n secondary extent size; 0 for none
    SITE TRACKS / CYLINDERS / BLOCKS the unit the two numbers are in; BLOCKS counts in BLKSIZE
    SITE UNIT= / SITE VOLUME= where it is placed — honoured since this release
    SITE RECFM= / LRECL= / BLKSIZE= as before

  • Unused space is released at close (#128). A data set that got CYL(10,10) and received 200 KB no longer keeps the ten cylinders. RLSE is what lets a site raise DEFPRIMARY without every small upload permanently occupying the generous default. It needed a library change — libc370#167, __txrlse() was present and never called — and applies only to data sets FTPD allocated itself.

Changed

  • Data set names without a version (#121). Rob Prins: "Please remove the mid qualifier of the data set names. E.g.: FTPD.LINKLIB is enough." They used to carry the full version — FTPD.V1R0M1.LINKLIB for 1.0.1, FTPD.V1R0M2.LINKLIB for 1.0.2 — while the FMID only moved with the minor release. So every patch landed in a fresh set of data sets that nothing pointed at, left the predecessor's standing, and needed the STC's STEPLIB re-edited before it became live at all. On the APF route it also needed a new IEAAPF00 entry and an IPL, every time. Now:

    FTPD.LINKLIB      the load module (APF this one, once, forever)
FTPD.SAMPLIB      the STC procedure and the configuration pattern
FTPD.AFTPDLOD     SMP's distribution library
FTPD.FTPDLOAD     staging; the install job scratches it when it is done

    The APF entry now needs one IPL ever, not one per release, because the entry names a data set that no longer changes.

  • TFTP110 deletes TFTP100, one FMID per release (#143) — see Why the first build was withdrawn above.

  • Out of space is a permanent failure, and says so (#129, #135). The reply is now 552, the code RFC 959 reserves for exceeded storage allocation, and it tells the client what to do about it:

    552 Requested file action aborted: out of space on the data set. Allocate
    more space with SITE PRIMARY/SECONDARY and CYLINDERS, or ask the
    administrator to raise DEFPRIMARY.

    A 552 is permanent: clients do not retry it. A genuine write error — bad track, device failure — still answers 451 with the byte count actually received, because that one is worth retrying. The two were one reply before.

  • The 256-byte receive cap is gone again (#124), because libc370#154 fixed the defect behind it in the library and 1.0.6 carries that fix. One cap is enough, and the library's is in the right place — see #122 below.

Fixed

  • An upload that ran out of space and lied about it (#127). Rob Prins reported an x37 ABEND during an upload, with the diagnosis attached: "It looks like that the allocation is too small. The maximum size is 85 tracks (TRK(10,5))." — and one more sentence that turned out to be the harder half: "And please don't retry after the abendx37." He was right on both counts, and the second one was the real defect. STOR allocated SPACE=TRK(10,5) with no way to change it, so an upload past ~85 tracks — about 4 MB at FB/80/3120 — hit an SB37/SD37/SE37. FTPD then answered 451, which RFC 959 defines as a transient condition: the client is entitled to retry, and a well-behaved one does. The retry met the same too-small data set, which was still catalogued, still holding the partial upload, and still exactly as full. Every attempt failed the same way, and the data set nobody could use stayed on DASD. Four changes came out of that: #100, #128, #129 and #135.

  • The failed upload no longer leaves a data set behind (#129). If FTPD allocated the data set for this transfer and the transfer failed, FTPD scratches it:

    FTPD074I SCRATCHED USER.UPLOAD.DATA AFTER A FAILED TRANSFER

    Only data sets FTPD created in that transfer — an upload into an existing data set is never scratched, whatever happens to it.

  • The recovery survives the close (#129). The first fix looked complete and was not. Under an SD37, fclose() never returned: stdio flushed the pending buffer into a DCB whose last write had already abended, and abended again inside the recovery routine. The recovery had already replied, so nothing was visibly wrong in the log — but the data connection was never closed, and the client sat waiting for a socket that would never shut down. Measured: 300 seconds to a client timeout, and the DD left allocated to the address space, so the scratch that followed got IDCAMS RC=8. The fix is an ordering — close the data connection before touching the FILE — plus two library changes: libc370#168 gave stdio a way to abandon a FILE whose last write failed (__fabandon()), and libc370#176 stopped stdio re-driving a WRITE on a dead DCB. With 1.0.6 the x37 no longer reaches FTPD as an ABEND at all: it arrives as ferror() with errno == ENOSPC, which is what #135 is about — the mechanism changed underneath the 552 and the reply had to move with it. The end-to-end test is in the suite: upload past the allocation, expect 552, expect the session to end promptly rather than in five minutes, expect the data set gone, and expect a retry from a fresh allocation to work.

  • Uploaded text arriving with lines missing (#122). meltdown03 uploaded a JCL member in ASCII mode and got it back damaged — not garbled characters, but missing lines, with a fragment of one line spliced onto another:

    000152 //SYSUT2   DD DSN=&&RXCAT,DISP=(,PASS),UNIT=VIO,
000153 //            SPAORACF TO REMOVE RACF INDICATOR      <<<< something bad happened here

    This is the X'75' restart defect, and it is not in FTPD. The socket read copies in 256-byte segments and the instruction is restartable; on a partial copy the count and the pointer disagree, and bytes are lost from the middle of the buffer. It only shows on receives larger than 256 bytes per call, which is why binary FB/80 transfers (80 bytes a call) never reproduced it and TYPE A transfers (4096) did. FTPD shipped a workaround first — cap every data-connection receive at one segment — and then removed it again in #124, because the library fixed it. Re-measured after the removal, TYPE A, 5 rounds of 2 MB: byte-identical every time. The send side is not fixed (#137) — see Still open.

  • DEFUNIT and DEFVOLUME did nothing (#133). Both were parsed, stored, dumped by /F FTPD,CONFIG — and never read by anything. SITE UNIT= and SITE VOLUME= were accepted and equally ignored. Every data set FTPD created landed wherever the system chose, which for most systems means a public pack rather than the one the administrator named. They are honoured now, so you can keep uploads off a system pack. Because a wrong value here is expensive, the configured ones are checked once at startup against the volumes actually online:

    FTPD059W DEFVOLUME=PUB009 IS NOT ONLINE -- NEW DATA SETS WILL BE PLACED BY THE SYSTEM
FTPD060W DEFUNIT=3350 DOES NOT MATCH DEFVOLUME=PUB000 (3390) -- DEFUNIT IGNORED

    A misconfigured volume is reported and ignored rather than failing every upload. A volume a client names with SITE VOLUME= is not silently corrected — but it is validated, because of what testing it turned up: SVC 99 asks the operator for an unmounted volume instead of failing. A single SITE VOLUME=NOVOL9 hung the session and left IEF238D outstanding on the console, waiting for a reply nobody knew to give. FTPD now checks the volume before allocating and sets S99NOMNT, so a bad name comes back as a reply to the client rather than a message to the operator. (libc370#181 tracks the library half.)

  • A UFS upload that failed still said it worked (#148). Found while relinking against ufsd 1.3.0, which fixed libufs reporting fewer bytes than a failed write had really moved — and FTPD turned out to be throwing the number away regardless. ftpd_ufs_stor() checked nothing. ufs_fwrite()'s count went unread, ufs_ferror() was never asked, and a data connection that broke left the receive loop exactly the way a completed transfer did. The reply was 226 Transfer complete in all three cases. A 226 is the client's cue to delete its copy — which, on a failed upload, is the only whole one left. There are three answers now:

    what happened reply
    out of space 552 — permanent, so a conforming client does not retry
    any other write failure the UFS error, mapped as elsewhere
    the data connection broke 426, per RFC 959
    clean end of data 226, as before

    This is the argument #129 made for MVS data sets, applied to the path that never got it. Expect the 552 more often than "disk full" suggests. UFS implements no double or triple indirect blocks, so a single file cannot exceed 16 direct plus 1024 single-indirect blocks — about 4.06 MB at the usual 4096-byte block size, on an otherwise empty filesystem. That is an ordinary size for a file someone uploads, and it is a per-file ceiling rather than a full volume, so the reply does not claim the filesystem is full. It is a documented property of the UFS on-disk format, not a bug in this release. Only the MVS side of STOR was covered before, and only since 1.1.0. UFS uploads have been answering 226 over truncated files since the UFS support went in.

Still open

The identity model is unchanged from 1.0.0, and so is the warning that came with it. The STC's resting identity is still STC/STCGROUP, which holds ALTER on every data set (#97); JES access still has no authorization model, JESOWNER being a client-set display filter rather than a control (#90); the ACEE switch is still not fully serialized against concurrent sessions (#64); and catalog calls still run under the STC identity rather than the session's (#123). Keep the FTPD identity scoped to what the started task itself needs — never give it broad data set authority.

The X'75' defect is only fixed on the receive side (#137). Every download that sends more than 256 bytes per call has the same exposure #122 had, and the library fix for it is still an open design question (libc370#160) — capping the send changes what every caller sees per call, so it needs measuring before patching.

The remaining known limits: a UFS create that fails for want of inodes still answers 452 although a full inode table lasts until the volume is reformatted (#151); shutdown can take up to ~5 seconds to notice idle sessions (#66); a NULL from __listpd()/__listds() is still read as an empty listing rather than checked for a storage shortage (#118); and a handful of z/OS SITE keywords are still missing (#130).

Thank you, Rob Prins and meltdown03 — three of the eleven issues in this release started as a report from outside and two came with the diagnosis already attached, and the report that caused the withdrawal (#141) came from someone installing the build on the day it shipped. #148 came out of comparing notes with the UFSD side while relinking against 1.3.0.

Full Changelog: v1.0.2...v1.1.0

Assets 5
Loading