(in directory: :filelink:`verification/tutorial_tracer_adjsens/`)
MITgcm has been adapted to enable AD using TAMC or TAF. The present description is specific to the use of TAMC or TAF as AD tool. The following sections describe the steps which are necessary to generate a tangent linear or adjoint model of MITgcm. We take as an example the sensitivity of carbon sequestration in the ocean. The AD-relevant hooks in the code are outlined in :numref:`sec_ad_tlm_and_adm` and :numref:`sec_ad_finalize_contribtuions`.
We describe an adjoint sensitivity analysis of out-gassing from the ocean into the atmosphere of a carbon-like tracer injected into the ocean interior (see Hill et al. 2004 :cite:`hill:04`).
For this work, MITgcm was augmented with a thermodynamically inactive tracer, C. Tracer residing in the ocean model surface layer is out-gassed according to a relaxation time scale, \mu. Within the ocean interior, the tracer is passively advected by the ocean model currents. The full equation for the time evolution
\frac{\partial C}{\partial t} \, = \, -U\cdot \nabla C \, - \, \mu C \, + \, \Gamma(C) \,+ \, S
also includes a source term S. This term represents interior sources of C such as would arise due to direct injection. The velocity term, U, is the sum of the model Eulerian circulation and an eddy-induced velocity, the latter parameterized according to Gent/McWilliams (Gent and McWilliams 1990 :cite:`gen-mcw:90`; Gent et al. (1995) :cite:`gen-eta:95`). The convection function, \Gamma, mixes C vertically wherever the fluid is locally statically unstable.
The out-gassing time scale, \mu, in :eq:`carbon_ddt` is set so that 1/\mu \sim 1 year for the surface ocean and \mu=0 elsewhere. With this value, :eq:`carbon_ddt` is valid as a prognostic equation for small perturbations in oceanic carbon concentrations. This configuration provides a powerful tool for examining the impact of large-scale ocean circulation on CO2 out-gassing due to interior injections. As source we choose a constant in time injection of S = 1 mol s-1.
The model configuration employed has a constant 4^\circ \times 4^\circ resolution horizontal grid and realistic geography and bathymetry. Twenty vertical layers are used with vertical spacing ranging from 50 m near the surface to 815 m at depth. Driven to steady-state by climatological wind-stress, heat and fresh-water forcing, the model reproduces well known large-scale features of the ocean general circulation.
To quantify and understand out-gassing due to injections of C in :eq:`carbon_ddt`, we define a cost function {\cal J} that measures the total amount of tracer out-gassed at each timestep:
{\cal J}(t=T)=\int_{t=0}^{t=T}\int_{A} \mu C \, dA \, dt
:eq:`cost_tracer` integrates the out-gassing term, \mu C, from :eq:`carbon_ddt` over the entire ocean surface area, A, and accumulates it up to time T. Physically, {\cal J} can be thought of as representing the amount of CO2 that our model predicts would be out-gassed following an injection at rate S. The sensitivity of {\cal J} to the spatial location of S, \frac{\partial {\cal J}}{\partial S}, can be used to identify regions from which circulation would cause CO2 to rapidly out-gas following injection and regions in which CO2 injections would remain effectively sequestered within the ocean.
The code customization routines are in :filelink:`verification/tutorial_tracer_adjsens/code_ad`:
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/COST_OPTIONS.h`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/CTRL_OPTIONS.h`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/CPP_OPTIONS.h`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/AUTODIFF_OPTIONS.h`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/CTRL_SIZE.h`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/GAD_OPTIONS.h`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/GMREDI_OPTIONS.h`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/SIZE.h`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/tamc.h`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/ctrl_map_ini_genarr.F`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/ptracers_forcing_surf.F`
- :filelink:`verification/tutorial_tracer_adjsens/code_ad/packages.conf`
The runtime flag and parameters settings are contained in :filelink:`verification/tutorial_tracer_adjsens/input/` and :filelink:`verification/tutorial_tracer_adjsens/input_ad/`, together with the forcing fields and and restart files:
- :filelink:`verification/tutorial_tracer_adjsens/input_ad/data`
- :filelink:`verification/tutorial_tracer_adjsens/input_ad/data.cost`
- :filelink:`verification/tutorial_tracer_adjsens/input_ad/data.ctrl`
- :filelink:`verification/tutorial_tracer_adjsens/input_ad/data.gmredi`
- :filelink:`verification/tutorial_tracer_adjsens/input_ad/data.grdchk`
- :filelink:`verification/tutorial_tracer_adjsens/input_ad/data.optim`
- :filelink:`verification/tutorial_tracer_adjsens/input_ad/data.pkg`
- :filelink:`verification/tutorial_tracer_adjsens/input_ad/data.ptracers`
- :filelink:`verification/tutorial_tracer_adjsens/input_ad/eedata`
verification/tutorial_tracer_adjsens/input/topog.bin
verification/tutorial_tracer_adjsens/input/windx.bin
,verification/tutorial_tracer_adjsens/inputwindy.bin
verification/tutorial_tracer_adjsens/input/salt.bin
,verification/tutorial_tracer_adjsens/input/theta.bin
verification/tutorial_tracer_adjsens/input/SSS.bin
,verification/tutorial_tracer_adjsens/input/SST.bin
Below we describe the customizations of this files which are specific to this experiment.
File :filelink:`code_ad/COST_OPTIONS.h /<verification/tutorial_tracer_adjsens/code_ad/COST_OPTIONS.h>`
This file contains package-specific CPP-options (see :numref:`pkg_cost_description`).
File :filelink:`code_ad/CTRL_OPTIONS.h /<verification/tutorial_tracer_adjsens/code_ad/CTRL_OPTIONS.h>`
This file contains package-specific CPP-options (see :numref:`sec:pkg:ctrl`).
File :filelink:`code_ad/CPP_OPTIONS.h /<verification/tutorial_tracer_adjsens/code_ad/CPP_OPTIONS.h>`
This file contains model-specific CPP options (see :numref:`customize_compilation`).
Most options are related to the forward model
setup. They are identical to the global steady circulation setup of
:filelink:`verification/global_ocean.90x40x15/`. The three options specific to
this experiment are as follows.
#define ALLOW_PASSIVE_TRACER
enables the code to carry through the advection/diffusion of
a passive tracer along the model integration.
#define ALLOW_MIT_ADJOINT_RUN
enables the inclusion of some AD-related fields concerning
initialization, link between control variables and forward model
variables, and the call to the top-level forward/adjoint subroutine
adthe_main_loop.F
instead of :filelink:`model/src/the_main_loop.F`.
#define ALLOW_GRADIENT_CHECK
enables the gradient check package. After computing the
unperturbed cost function and its gradient, a series of computations
are performed for which:
- an element of the control vector is perturbed
- the cost function w.r.t. the perturbed element is computed
- the difference between the perturbed and unperturbed cost function is computed to compute the finite difference gradient
- the finite difference gradient is compared with the adjoint-generated gradient.
The gradient check package is further described in :numref:`ad_gradient_check`.
The CPP options of several AD-related packages are grouped in this file:
- Overall ECCO-related execution modus:These determine whether a pure forward run, a sensitivity run or an iteration of optimization is performed. These options are not needed in the present context.
- Adjoint support package: :filelink:`pkg/autodiff/`This package contains hand-written adjoint code such as active file handling, flow directives for files which must not be differentiated, and TAMC-specific header files.
#define
:varlink:`ALLOW_AUTODIFF_TAMC` defines TAMC-related features in the code.#define
:varlink:`ALLOW_TAMC_CHECKPOINTING` enables the checkpointing feature of TAMC (see :numref:`sec_autodiff_storage_v_recompute`). In the present example a 3-level checkpointing is implemented. The code contains the relevant store directives, common block and tape initializations, storing key computation, and loop index handling. The checkpointing length at each level is defined in file :filelink:`code_ad/tamc.h <verification/tutorial_tracer_adjsens/code_ad/tamc.h>`, see :ref:`below <tut_tracer_adjsens_tamc>`. The out and intermediate loop directives are contained in the files :filelink:`pkg/autodiff/checkpoint_lev3_directives.h`, :filelink:`pkg/autodiff/checkpoint_lev2_directives.h`.#define
:varlink:`ALLOW_AUTODIFF_MONITOR` enables the monitoring of intermediate adjoint variables (see :numref:`sec_autodiff_output_adj_vars`).#define
:varlink:`ALLOW_DIVIDED_ADJOINT` enables adjoint dump and restart (see :numref:`sec_autodiff_diva`). - Cost function package: :filelink:`pkg/cost/`This package contains all relevant routines for initializing, accumulating and finalizing the cost function (see :numref:`pkg_cost_description`).
#define
:varlink:`ALLOW_COST` enables all general aspects of the cost function handling, in particular the hooks in the forward code for initializing, accumulating and finalizing the cost function.#define
:varlink:`ALLOW_COST_TRACER` includes the call to the cost function for this particular experiment, eqn. :eq:`cost_tracer`. - Control variable package: :filelink:`pkg/ctrl/`This package contains all relevant routines for the handling of the control vector. Each control variable can be enabled/disabled with its own flag:
#define
:varlink:`ALLOW_THETA0_CONTROL`initial temperature
#define
:varlink:`ALLOW_SALT0_CONTROL`initial salinity
#define
:varlink:`ALLOW_TR10_CONTROL`initial passive tracer concentration
#define
:varlink:`ALLOW_TAUU0_CONTROL`zonal wind stress
#define
:varlink:`ALLOW_TAUV0_CONTROL`meridional wind stress
#define
:varlink:`ALLOW_SFLUX0_CONTROL`freshwater flux
#define
:varlink:`ALLOW_HFLUX0_CONTROL`heat flux
#define
:varlink:`ALLOW_DIFFKR_CONTROL`diapycnal diffusivity
isopycnal diffusivity
.. literalinclude:: ../../../verification/tutorial_tracer_adjsens/code_ad/SIZE.h :linenos: :caption: verification/tutorial_global_oce_latlon/code/SIZE.h
The file contains the grid point dimensions of the forward model. It is identical to the :filelink:`verification/exp2/`.
This file contains common blocks of some adjoint variables that are generated by TAMC. The common blocks are used by the adjoint support routine :filelink:`/pkg/autodiff/addummy_in_stepping.F` which needs to access those variables:
common /:varlink:`addynvars_r`/ | is related to :filelink:`model/inc/DYNVARS.h` |
common /:varlink:`addynvars_cd`/ | is related to :filelink:`model/inc/DYNVARS.h` |
common /:varlink:`addynvars_diffkr`/ | is related to :filelink:`model/inc/DYNVARS.h` |
common /:varlink:`addynvars_kapgm`/ | is related to :filelink:`model/inc/DYNVARS.h` |
common /:varlink:`adtr1_r`/ | is related to TR1.h |
common /:varlink:`adffields`/ | is related to :filelink:`model/inc/FFIELDS.h` |
Note that if the structure of the common block changes in the above
header files of the forward code, the structure of the adjoint common
blocks will change accordingly. Thus, one must make sure that the
structure of the adjoint common block in the hand-written file
:filelink:`/pkg/autodiff/adcommon.h` complies with the automatically generated adjoint common
blocks in adjoint_model.F
. The header file is enabled via the
CPP-option :varlink:`ALLOW_AUTODIFF_MONITOR`.
This routine contains the dimensions for TAMC checkpointing and some indices relevant for storing ky computations.
- 3-level checkpointing is enabled, i.e., the timestepping is divided into three different levels (see :numref:`sec_autodiff_storage_v_recompute`). The model state of the outermost (:varlink:`nchklev_3`) and the intermediate (:varlink:`nchklev_2`) timestepping loop are stored to file (handled in :filelink:`model/src/the_main_loop.F`). The innermost loop (:varlink:`nchklev_1`) avoids I/O by storing all required variables to common blocks. This storing may also be necessary if no checkpointing is chosen (nonlinear functions, if-statements, iterative loops, ...). In the present example the dimensions are chosen as follows:
nchklev_1 = 36 nchklev_2 = 30 nchklev_3 = 60
To guarantee that the checkpointing intervals span the entire integration period the following relation must be satisfied:where :varlink:`nTimeSteps` is either specified in :filelink:`input_ad/data <verification/tutorial_tracer_adjsens/input_ad/data>` or computed via: - No checkpointing is enabled. In this case the relevant counter is :varlink:`nchklev_0`. Similar to above, the following relation has to be satisfied:
The following parameters may be worth describing: :varlink:`isbyte`, :varlink:`maxpass`.
This file contains all relevant parameter flags and lists to run TAMC or
TAF. It is assumed that TAMC is available to you, either locally, being
installed on your network, or remotely through the ’TAMC Utility’. TAMC
is called with the command tamc
followed by a number of options. They
are described in detail in the TAMC manual (Giering 1999 :cite:`giering:99`).
Here we briefly discuss the main flags used in the makefile
. The
standard output for TAF is written to file taf.log
.
TAMC:
-input «variable names» -output «variable name» -i4 -r4 ... -toplevel «S/R name» -reverse «file names»
TAF:
-input «variable names» -output «variable name» -i4 -r4 ... -toplevel «S/R name» -reverse «file names» -flow taf_flow.log -nonew_arg
-toplevel «S/R name»
Name of the toplevel routine, with respect to which the control flow analysis is performed.
input «variable names»
List of independent variables u with respect to which the dependent variable J is differentiated.
-output «variable name»
Dependent variable J which is to be differentiated.
-reverse «file names»
Adjoint code is generated to compute the sensitivity of an independent variable w.r.t. many dependent variables. In the discussion of :numref:`chap_autodiff` the generated adjoint top-level routine computes the product of the transposed Jacobian matrix M^T times the gradient vector \nabla_v J. «file names» refers to the list of files
.f
which are to be analyzed by TAMC. This list is generally smaller than the full list of code to be compiled. The files not contained are either above the top-level routine (some initializations), or are deliberately hidden from TAMC, either because hand-written adjoint routines exist, or the routines must not (or don’t have to) be differentiated. For each routine which is part of the flow tree of the top-level routine, but deliberately hidden from TAMC (or for each package which contains such routines), a corresponding file.flow
exists containing flow directives for TAMC.-i4 -r4
-flow taf_flow.log
Will cause TAF to produce a flow listing file named
taf_flow.log
in which the set of active and passive variables are identified for each subroutine.-nonew_arg
The default in the order of the parameter list of adjoint routines has changed. Before TAF 1.3 the default was compatible with the TAMC-generated list. As of TAF 1.3 the order of adjoint routine parameter lists is no longer compatible with TAMC. To restore compatibility when using TAF 1.3 and higher, this argument is needed. It is currently crucial to use since all hand-written adjoint routines refer to the TAMC default.
Contains 2-D bathymetry information.
Files input/windx.bin
, input/windy.bin
, input/salt.bin
, input/theta.bin
, input/SSS.bin
, input/SST.bin
These contain the initial values of salnity and potential temperature (salt.bin
, theta.bin
), surface boundary values (surface wind stresses
windx.bin
, windy.bin
), and surface restoring fields (SSS.bin
, SST.bin
).
The build process of the adjoint model is slightly more complex than
that of compiling the forward code. The main reason is that the adjoint
code generation requires a specific list of routines that are to be
differentiated (as opposed to the automatic generation of a list of
files to be compiled by :filelink:`genmake2 <tools/genmake2>`). This list excludes routines that don’t
have to be or must not be differentiated. For some of the latter
routines flow directives may be necessary, a list of which has to be
given as well. For this reason, a separate makefile
is currently
maintained in the directory adjoint/. This makefile is responsible for
the adjoint code generation.
In the following we describe the build process step by step, assuming you are in the directory bin/. A summary of steps to follow is given at the end.
ln -s ../verification/???/code/.genmakerc .
ln -s ../verification/???/code/*.[Fh] .
Link your customized genmake options, header files, and modified code to the compile directory.../tools/genmake -makefile
Generate your Makefile (see :numref:`genmake2_desc`).make depend
Dependency analysis for the CPP pre-compiler (see :numref:`building_quickstart`).cd ../adjoint
make adtaf
ormake adtamc
Depending on whether you have TAF or TAMC at your disposal, you’ll choose adtaf or adtamc as your make target for themakefile
in the directory adjoint/. Several things happen at this stage.
make adrestore
make ftlrestore
The initial template filesadjoint_model.F
andtangentlinear_model.F
in :filelink:`pkg/autodiff` which are part of the compiling list created by :filelink:`genmake2 <tools/genmake2>` are restored.make depend, make small_f
The bin/ directory is brought up to date, i.e., for recent changes in header or source code.[Fh]
, corresponding.f
routines are generated or re-generated. Note that here, only CPP pre-compiling is performed; no object code.o
is generated as yet. Pre-compiling is necessary for TAMC to see the full code.make allcode
All Fortran routines .f in bin/ are concatenated into a single file calledtamc_code.f
.make admodeltaf/admodeltamc
Adjoint code is generated by TAMC or TAF. The adjoint code is written to the filetamc_code_ad.f
. It contains all adjoint routines of the forward routines concatenated intamc_code.f
. For a given forward routine subroutine routinename the adjoint routine is named adsubroutine routinename by default (that default can be changed via the flag-admark «markname»
). Furthermore, it may contain modified code which incorporates the translation of adjoint store directives into specific Fortran code. For a given forward routines subroutine routinename the modified routine is named mdsubroutine routinename. TAMC or TAF info is written to filetamc_code.prot
ortaf.log
, respectively.make adchange
The multi-threading capability of MITgcm requires a slight change in the parameter list of some routines that are related to to active file handling. This post-processing invokes the sed script :filelink:`tools/adjoint_sed` to insert the threading counter :varlink:`myThId` into the parameter list of those subroutines. The resulting code is written to filetamc_code_sed_ad.f
and appended to the fileadjoint_model.F
. This concludes the adjoint code generation.
cd ../bin
make
The fileadjoint_model.F
cnow contains the full adjoint code. All routines are now compiled.
N.B.: The targets make adtaf/adtamc
now comprise a series of targets
that in previous versions had to be invoked separately. This was
probably preferable at a more experimental stage, but has now been
dropped in favor of a more straightforward build process.
cd bin ln -s ../verification/my_experiment/code/.genmakerc . ln -s ../verification/my_experiment/code/*.[Fh] . ../tools/genmake -makefile make depend cd ../adjoint make adtaf <OR: make adtamc> contains the targets: adrestore small_f allcode admodeltaf/admodeltamc adchange cd ../bin make