INSTALL

BUILD AND INSTALL HPSS
======================

Building HPSS is beyond the scope of this guide but you should have a working 
HPSS installation. This version of the DSI was tested against HPSS 7.4.2. The
DSI requires either a full HPSS build or a 'clnt' HPSS build. Note whether the
HPSS build is 32bit or 64bit; you'll need to build GridFTP and the DSI that same
way.

POINTS OF INTEREST ON THIS VERSION
==================================
* This DSI is likely to have 64bit-ism and so 64bit is recommended. 
* This DSI requires UDA support in HPSS for checksums and quotas.
* The format of the stored checksums are compatible with 7.4.2 but
  not necessarily with earlier versions.
* BZ2487 is a show stopper; you must have it
* The GridFTP client you choose must use ALLO (always and corretly)
* Checkout the README.xdr if you want to understand why the DSI
  loading sequence is setup as it is
* Because of the XDR issue, this DSI most likely only works on Linux
* Restarts are not supported until Globus supplies a way to decouple
  restart from performance markers. See http://jira.globus.org/browse/GT-517.


CONFIGURATION OPTIONS
=====================
* --enable-quota_support puts a special UDA mark on files that are renamed. This
  funtionality is part of the NCSA HPSS quota system. There are other parts of
  the quota system that are not distributed publically that make this funciton.
  If you would like to have that code, you can contact me. Otherwise, do not
  enable this feature.

REQUIRED HPSS PATCHES
=====================
BZ2487 - PIO returns wrong bytes moved when reading without regard to file 
         length, also returns incorrect data. The gap handling code may be
         erroneously adding the gap offset to the total bytes moved, or at
         least adding additional bytes when handling gap information. This
         bug may not exist without the patch for BZ2146. The fix is available
         for 7.4.1, not 7.3.3.

RECOMMENDED HPSS PATCHES
========================
BZ2146 - PIO with HPSS_PIO_HANDLE_GAP does not properly handle 32 IOD SrcSink 
         descriptor boundaries. HPSS breaks transfers into 32 segments per
         transfer (IOD/IOR combos). If you do not specify HPSS_PIO_HANDLE_GAP,
         hpss_PIOExecute() will kickout similar to gap handling; it is up to
         the client to reissue the hpss_PIOExecute() call with the necessary
         offset and length. When HPSS_PIO_HANDLE_GAP is used, hpss_PIOExecute()
         will treat this segment-limit-kickout as a gap and resume the transfer
         without the kickout. However, this bug causes hpss_PIOExecute() to get
         stuck in an infinite loop.

         This bug has been fixed in HPSS 7.4.1 but not in 7.3.3. The portable
         solution is to not use HPSS_PIO_HANDLE_GAP and have the DSI handle
         the gaps. Note that if a site uses the "Fixed Length, Classic Style"
         allocation scheme (which is recommended for GridFTP to get optimal
         performance) then this bug is not encountered because the segment
         count remains low (4).

         This DSI is currently not affected by this bug because it does not use
         HPSS_PIO_HANDLE_GAP.

BZ2819 - PIO 60 second delay impacts small file performance. There is a small
         percentage chance that, after a transfer completes, HPSS PIO will
         wait 60 seconds before informing the client that the transfer has
         completed. This fix has been implemented in 7.4.1 and will be ported
         to 7.3.3.

HPSS BUGS TO MONITOR
====================
BZ1660 - PIO callback buffer switching does not persist across calls to 
         hpss_PIOExecute. The buffer passed to hpss_PIORegister() is passed
         to it's callback with a pointer-to-pointer variable. It seesm that
         the buffers can be swapped from call to call however, each time
         hpss_PIOExecute() is called (which happends multiple times per 
         transfer for files with gaps or large segment counts) the buffer
         passed to hpss_PIORegister() is passed to the callback on its first
         invocation after each call to hpss_PIOExecute(). If buffers have been
         swapped, the callback will receive a buffer that may be currently
         used else where. The fix is to avoid buffer swapping.

         IBM HPSS has decided that this is intended behavior and so the DSI works
         around the issue.

BZ2856 - Enabling HPSS_API_REUSE_CONNECTIONS returns address already in use.
         This one sets a limit on how many active connections we can have.
         GridFTP and HPSS are both socket hogs. Quick, successive file transfers
         can lead the system to run out of available ports. There is no fix for
         this bug at this time. The number of ephemeral ports can be increased
         and the amount of time a socket spends in timed wait can be decreased
         to help avoid this issue.

PERFORMANCE
===========
(Short version) GridFTP installations benefit from and take full advantage of
classes of service that use fixed length classic style allocation. In short,
you'll get the best performance from the GridFTP interface (actually any HPSS
interface) if the segment count is below 32.

(Longer version) HPSS has multple disk/tape allocation algorithms used to
allocate space for incoming data. Fixed length allocation gives you equal size
chunks to store data in. This was deemed wasteful because the last block was
most certainly never filled. Variable length allocation was created to solve this
problem; it will give you increasingly larger segments as data is stored and
truncates the last block. This is a win for most situations when HPSS is unsure
how much data is to be stored for the given file.

Using either of these allocation mechanisms (any variable length allocation or
fixed w/o knowing the file size), HPSS is free to continue to allocate segments
until all the data is stored. This has a definite performance impact because
internally HPSS retrieves data in 32-segment chunks. This means when you request
a file from HPSS, internally it breaks it up into multiple transfers, each of which
is <= 32 segments. Functionally, this is transparent to the client. In terms of 
performance, the client will see a high load followed by a pause followed by a
high load, etc.

In order to avoid the performance hit, you can use fixed length allocation with
segment counts < 32 and take advantage of the fact that any WELL-BEHAVED GridFTP
client will inform HPSS of the size of the incoming file before the transfer
begins. In fact, the DSI is designed to require this. If a GridFTP client is
not well behavad, the DSI will act as though a zero length transfer is about to 
occur and will handle it as such. So you'll know if the client is not doing the 
right thing.

GRIDFTP PATCHES
===============
These are fixed in Globus Toolkit 5.2.4:
  GT-297 globus_ftp_control_data_query_channels() SIGSEGV on proxy expiration

Still believed to exist in GT 5.2.5:
  GT-296 globus_ftp_control_data_read() race condition
  GT-376 globus_ftp_control_data_query_channels() SIGSEGV on ABRT
  GT-377 abort() in globus_ftp_control_handle_destroy() -> globus_list_remove()
  GT-378 abort() in globus_i_gfs_data_session_stop()
  GT-379 Abort() in globus_i_gfs_data_request_recv()
  GT-380 Abort in globus_l_gfs_new_server_cb()

INSTALL GRIDFTP
===============

In order to build the module, you'll need to install the globus-gridftp-server-devel
and libgssglue-devel rpms. In order to run the HPSS GridFTP server, you'll need to
install the globus-gridftp-server RPM. This DSI is only tested against 64 bit builds.

See the GT documentation for installing from RPM.

http://toolkit.globus.org/toolkit/docs/5.2/5.2.5/admin/install/#install-bininst

BUILD AND INSTALL THE DSI
=========================

Build the HPSS DSI with the following options. Set the value of '--libdir' to the
directory where you want to place the DSI. For your sanity, I recommend putting it
in a new directory with the version information stored in the name. 
Ex. /usr/local/gridftp_hpss_dsi-1.0. You can then set the environment variable
LD_LIBRARY_PATH to this location so that the GridFTP server can find the DSI.

If you use Globus RPMs, you shoudn't need to specify --with-globus because it 
defaults to /usr/

./configure --with-hpss=/opt/hpss -libdir=/usr/local/gridftp_hpss_dsi-<version>.

Install the DSI into libdir:

make install

USE USER HPSSFTP FOR GRIDFTP
============================
GridFTP requires a privileged user with control permission on the core server's
client interface  in order to log into HPSS and then it changes its credentials
to that of the actual user (you). Think of it as a type of setuid process that
logs in, does what it must, then changes the process owner to you.

In the past, we recommended creating a new HPSS user named 'gridftp' for this
purpose. However, it was discovered that HPSS allows any connection with control
permission on the core server's client interface to bypass Gate Keeper site hooks.
That little tidbit breaks site specific accounting and quota deployments. 

HPSS has a special privileged user named 'hpssftp' which has the necessary permissions
and special code within HPSS to not allow it to bypass the Gate Keeper callouts. Thus
we recommend that you use the 'hpssftp' user in gridftp.conf.

You will however want the GridFTP process to run as a non privileged user. For this
reasons, we run the GridFTP server as local UNIX user 'gridftp' but configure it to
use hpssftp's credentials to log into HPSS.

Add the new user 'gridftp' to the system password file. Your entry should look
something like the following:

gridftp:x:316:316:GridFTP Server:/home/gridftp:/bin/bash

Now you'll need to create a keytab file for the gridftp user. This will be used
by the DSI in order to authenticate to HPSS. For sites using unix
authentication with HPSS (rather than kerberos):

[root@hpss gridftp_hpss_dsi_0_1]# /opt/hpss/bin/hpss_unix_keytab -f /var/hpss/etc/gridftp.keytab add hpssftp

For sites using kerberos authentication with HPSS, you'll need to create
and use a kerberos keytab file (rather than a unix keytab). The kerberos
utility 'ktutil' can be used for that purpose.

CONFIGURING XINETD
==================

The globus gridftp server does not need to run as a privileged user. You should
create a system account for this purpose. It is important to make sure that both
the keytab files generated above and the GSI host key are both owned by this
system account. In the following example, I will use the system account
'gridftp'. Since our HPSS installation currently uses the system password file
(instead of the HPSS specific password file) this is the same account created
earlier for authentication to HPSS. However, if your site uses the HPSS specific
password file, you will need to create a separate system account for the GridFTP
server process (they may both have the same name).

Add an entry to /etc/services for the default gsiftp port:

gsiftp 2811/tcp # GSI FTP

Here's an example xinetd entry (/etc/xinetd.d/gridftp). Note that the DSI
requires that the server run with threads so you must use -threads with a value
of 2 or greater.

service gsiftp

{
	flags       = IPv4
	wait        = no
	user        = gridftp
	# You may need to set group depending upon the account chosen
	group       = gridftp
	server      = /usr/local/globus/sbin/globus-gridftp-server
	# auth-level 6 = 2 for frontend default + 4 for no-change-uid/gid
	#  0 - Disables all authorization checks
	#  1 - Authorize identity
	#  2 - Authorize all file/resource accesses (Default for fronend)
	#  4 - Disable changing process uid to authenticated user
	server_args = -inetd -threads 2 -auth-level 7 -dsi hpss_local -disable-command-list SCKS,APPE,REST
	env         = GRIDMAP=/etc/grid-security/grid-mapfile
	env         = LD_LIBRARY_PATH=/opt/hpss/lib:/usr/local/gridftp_hpss_dsi-<version>
	socket_type = stream
	per_source  = 100
}

CONFIGURING WITHOUT XINETD
==========================

If you are launching GridFTP without xinetd (for example, a Globus Connect Server install),
you can configure the server as follows.

Add these lines to /etc/gridftp.d/extra:
	auth_level 7
	load_dsi_module hpss_local
	disable_command_list SCKS,APPE,REST

Also, in order for the GridFTP server to find the HPSS DSI, you must edit your start script
to include: LD_LIBRARY_PATH="<path to DSI install dir>". For example, if you launch GridFTP
from /etc/init.d/globus-gridftp-server and the DSI is installed in /usr/local/hpss_dsi, then
add the following line to /etc/init.d/globus-gridftp-server:
  export LD_LIBRARY_PATH=/usr/local/hpss_dsi

SETUP THE DSI CONFIG FILE
=========================

Review 'ConfigFile' in the source directory for any changes you may wish to
make for your site. Then copy ConfigFile into place on the target system. The
DSI will use the following search order for locating the configuration file:

 1) $HPSS_PATH_ETC/gridftp.conf
 2) /var/hpss/etc/gridftp.conf

Make sure the configuration file's permissions allow for the GridFTP process to
read it. For example, if you chose to run the GridFTP server as user 'gridftp',
make sure the user 'gridftp' has read access to the configuration file.

HPSS Configuration Files
========================
The HPSS DSI needs to run on a system that has sufficient HPSS configuration files installed
to permit it to talk to the appropriate HPSS servers (such as an HPSS mover node), as well as
to perform authentication of users.  These files typically are kept under /var/hpss/etc.
The HPSS user authentication files may be separate from the system's authentication files
(i.e. in /var/hpss/etc/{group,passwd}), but can be set in /var/hpss/etc/env.conf to point
to any suitable files.  The DSI will also need to store credential files in the /var/hpss/cred
directory.

Kerberos Configuration
======================
Kerberos must be configured for access to the proper Kerberos realm that contains HPSS.
This file is usually kept in /etc/krb5.conf.  You may need to enable the allow_weak_crypto
option in the [libdefaults] section if the DSI module can not talk to the HPSS servers.