Skip to content

Tokens & Variables

Vince LaGrassa edited this page Mar 21, 2025 · 18 revisions

Tokens

Tokens are changing values that may be used in strings. They are typically used in filenames / filepaths, to distinguish e.g. between data files for different species or different releases.

For example, some filepaths may be given in the format:

... /{ SPECIES }/{ RELEASE }/sample_file_name-{ SPECIES }-{ RELEASE }.txt

If we wanted to use the version of this file associated with C. elegans from the (made-up) CaeNDR release on Jan 1, 2000, we would look for the file:

... /c_elegans/20000101/sample_file_name-c_elegans-20000101.txt

Token List

Token Description Values
SPECIES The relevant Caenorhabditis species for a given dataset or operation. The (unofficial) format is genus initial + underscore + species name -- in our case, this means all valid values begin with c_. c_elegans, c_briggsae, c_tropicalis
RELEASE The CaeNDR release that a data file was released in. New values added when new dataset releases are created. "YYYYMMDD" format
SVA The CaeNDR release to use for the Strain Variant Annotation table & tool. This has the same format (and possible values?) as the RELEASE token, but is kept separate because this tool may lag behind the most recent CaeNDR release - i.e. the SVA release and the "main" release may be different. "YYYYMMDD" format, same as RELEASE
GENOME An attached to a given release, relevant for locating the correct FASTA file. I believe this is now a more generic version of the WB token, since CaeNDR expanded from using specifically WormBase data to also using lab-produced genomes. Typically 1-to-1 with RELEASE values, but (a) it is possible for multiple CaeNDR releases to use the same genome, and (b) the genome value may be lab-internal or lab-specific, i.e. have some meaning within the lab that isn't captured by the CaeNDR release value alone. Could be anything - WS276 (from WormBase), Feb2020 (from Andersen Lab), etc.
STRAIN An identifier for a specific strain. Mostly relevant for the Genome Browser tool's IGV Browser element, which needs to pull files for specific strains. BRC20067, etc.
USER_ID The unique internal ID of a CaeNDR user who submitted a given data file, e.g. a phenotype trait file. GCP Datastore Key
PRJ WormBase project number associated with a dataset or release. Pulled from WormBase. May not be relevant going forward, with new CaeNDR release format. PRJNA13758, etc.
WB WormBase version associated with a dataset or release. Pulled from WormBase. May not be relevant going forward, with new CaeNDR release format. WS276, etc.

A few notes:

  • Probably 85~90% of the time, you only need to care about the SPECIES and RELEASE tokens. All the others are pretty niche and context-specific; the SVA token, for example, is pretty much only relevant to the Strain Variant Annotation tool.
  • The SVA token may be referred to with RELEASE, if the Strain Variant Annotation versioning scheme isn't relevant from a particular perspective. (This can be a bit confusing, but also, it's pretty infrequent that we actually care about this difference.)
  • The GENOME token value is relevant when creating a new dataset release, and will appear on the Dataset Release page. It's mostly used to associate lab-internal versioning with CaeNDR release versioning. It's possible for multiple releases to use the same GENOME value, if they're based on the same FASTA genome file.
  • The tokens PRJ and WB are holdovers from an older dataset versioning system, and don't appear to be very relevant going forward. It may be helpful to know what they meant, though, if dealing with old data.

Site Environment Variables

This is NOT an extensive list of all environment variables required to use the site! Rather, this is an outline of most of the "custom" environment variables defined & used by the CaeNDR source code, specifically for running the site.

All environment variables here are directly read & used in the source code itself.

Module Configuration

Variable Description Type
MODULE_{NAME}_CONTAINER_NAME The name of the Docker container to look for to access this module. string
MODULE_{NAME}_CONTAINER_VERSION The version tag to look for on the Docker container. string

Note that the Image Thumbnail Generator module uses a slightly different format:

Variable Description Type
MODULE_IMG_THUMB_GEN_SOURCE_PATH The path in GCP where images are held. string
MODULE_IMG_THUMB_GEN_VERSION The version tag to look for on the Docker container. (Same as above) string

Containerized Tool Configuration

See (tools section)

GCP Bucket Names

See (buckets page?)

Variable Description Type
MODULE_SITE_BUCKET_PHOTOS_NAME The "photos" bucket name. string
MODULE_SITE_BUCKET_ASSETS_NAME The "assets" bucket name. string
MODULE_SITE_BUCKET_PRIVATE_NAME The "private" bucket name. string
MODULE_SITE_BUCKET_PUBLIC_NAME The "public" bucket name. string
MODULE_DB_OPERATIONS_BUCKET_NAME The "db" bucket name. string
MODULE_SITE_BUCKET_DATASET_RELEASE_NAME ... string

Override:

Variable Description Type
MODULE_SITE_BUCKET_PUBLIC_NAME_OVERRIDE ... string

Filepaths & Filenames

All(?) filepaths and filenames are handled as "tokenized strings", i.e. strings that may contain one or more "Token" values (see above).

BAM/BAI Files

Variable Description File Type Type
BAM_BAI_PREFIX The location of the BAM and BAI files within the private bucket. - string (tokenized)
BAM_BAI_DOWNLOAD_SCRIPT_NAME The filename to download the "download BAM/BAI files" script with; that is, the filename that the user will see when they generate this file & download it. Bash (.sh) string (tokenized)

FASTA Files

Variable Description Variable Type
FASTA_FILENAME_TEMPLATE Name of the FASTA file for a given species & release, NOT including the file extension. This is because the full file and the index file share the same name. string (tokenized)
FASTA_EXTENSION_FILE The filename extension for the full FASTA file. Typically .fa. string
FASTA_EXTENSION_INDEX The filename extension for the FASTA index file. Typically .fa.fai. Note - NOT appended to the plain file extension, so they can be different. string

SQL Table Source Files

Files used primarily to build the SQL database tables. These are all read as tokenized strings, in case the filenames need to change for species / release / etc, but as of March 2025, only the SVA_CSVGZ file actually uses any tokens.

Variable Description File Type Type
GENE_GFF_FILENAME Name of the file used to build the "Wormbase Gene Summary" table. Name only - path is set elsewhere. Zipped GFF (.gff3.gz) string (tokenized)
GENE_GTF_FILENAME Name of one file used to build the "Wormbase Gene" table. Name only - path is set elsewhere. Zipped GTF (.gtf.gz) string (tokenized)
GENE_IDS_FILENAME Name of one file used to build the "Wormbase Gene" table. Name only - path is set elsewhere. Zipped Text (.txt.gz) string (tokenized)
SVA_CSVGZ_FILENAME Name of the file used to build the "Strain Variant Annotation" table. Name only - path is set elsewhere. Zipped CSV (.csv.gz) string (tokenized)

Miscellaneous Files

Other files used to populate the site.

Variable Description File Type Type
EULA_FILE_NAME The file containing the site's End-User License Agreement. Markdown (.md) string

Misc

Variable Description Type
MODULE_SITE_PASSWORD_PROTECTED Whether to request a password to access the site at all. Relevant for QA site. boolean
MODULE_SITE_CART_COOKIE_NAME Name for the cookie to use to store the user's cart, if they are requesting strains. string
MODULE_SITE_CART_COOKIE_AGE_SECONDS Timeout until the cart cookie expires, in seconds. int
MODULE_SITE_PASSWORD_RESET_EXPIRATION_SECONDS Timeout until a "password reset" link expires, in seconds. int
USER_OWNED_ENTITY_CACHE_AGE_SECONDS Timeout for the local cache of User entities, specifically when loading entities that are owned by users. Makes big queries where many objects have the same users much more efficient (e.g. pulling up all of a user's generated reports). int

Containerized Tool Variables

...

Google Cloud Platform Variables

Variables used to configure GCP access. See GCP docs for details.

GCP Project Configuration

Token Type
GOOGLE_CLOUD_PROJECT_ID string
GOOGLE_CLOUD_PROJECT_NUMBER string
GOOGLE_CLOUD_REGION string
GOOGLE_CLOUD_ZONE string
GOOGLE_CLOUD_APP_LOCATION string

Google Datastore

Token Type
GOOGLE_STORAGE_SERVICE_ACCOUNT_NAME string

Google SQL

Token Type
GOOGLE_CLOUDSQL_SERVICE_ACCOUNT_NAME string

Google Analytics

Token Type
GOOGLE_ANALYTICS_SERVICE_ACCOUNT_NAME string
GOOGLE_ANALYTICS_PROPERTY_ID string

Google Sheets

Token Type
GOOGLE_SHEETS_SERVICE_ACCOUNT_NAME string

Clone this wiki locally