PGNC External Stack

Overview

This repository contains the external technology stack for the PGNC (Plant Gene Nomenclature Committee) website and database. The PGNC provides standardized gene nomenclature for plant species, serving as a central resource for plant genomics research.

The stack is a containerized microservices architecture that provides a complete web application for plant gene data management, search, and visualization.

Documentation Index

• Angular docs hub: angular/angular.md
• Angular source overview: angular/src/src.md
• Python docs hub: python/docs/README.md
• SSL renewal setup: SSL_RENEWAL_SETUP.md
• Changelog: CHANGELOG.md

Documentation Quality Checks

To validate docs locally:

# Markdown style/lint
npm run lint:md

# Markdown links (ignores localhost, shields badges, and coverage output)
npm run lint:links

# Run both
npm run lint:docs

Architecture

The project relies on several microservices/components organized as Docker containers:

Frontend & API

• pgnc-ext-angular: Angular frontend application for the website. See Angular Documentation for technical details.
• pgnc-api: NestJS REST API providing backend services for the website and public API endpoints.

Search & Indexing

• pgnc-solr: Apache Solr search engine for fast gene data retrieval with BasicAuth security enabled.
• pgnc-solr-client: Server-side client that provides a secure interface between the frontend and Solr.
• pgnc_solr_load: Initial data loading service that populates Solr with indexed gene data from the database.
• python/: Python utilities for data processing and Solr index management, including:
  • Data loading and updating scripts
  • Gene model definitions
  • Comprehensive test suites

Data & Infrastructure

• pgnc_db_schema: PostgreSQL database schema and initial data (gzipped).
• pgnc-ext-solr-data: Persistent volume for Solr search indices.
• pgnc-ext-nginx: Reverse proxy and load balancer for external-facing components.
• pgnc-certbot: SSL certificate management using Let's Encrypt.

Prerequisites

• Container Runtime: Docker, Podman, or OrbStack for building and running containers
• Git: For cloning the repository and submodules
• Google Cloud Platform Access (for SSL):
  • gcp-key.json: Service account key file with Cloud DNS access
  • Place in the certbot/ directory for SSL certificate management
• Environment Configuration:
  • .env file: Copy sample.env to .env and configure for your environment
  • Contains database credentials, API keys, Solr authentication credentials, and service configurations

Quick Start

Method 1: Automated Setup (Recommended)

The total-refresh.sh script provides automated environment setup and management for the PGNC stack. This is the recommended approach for both new installations and regular maintenance.

New Environment Setup

# Clone the repository
git clone --recursive https://github.com/HGNC/pgnc-external-stack.git
cd pgnc-external-stack

# Configure environment variables
cp sample.env .env
# Edit .env with your specific configuration
# You could also cp a .env into the root of the project

# If using SSL add your gcp-key.json to certbot
cp ../gcp-key.json certbot/gcp-key.json

# Set up new environment with Docker
./total-refresh.sh --new --container-tool docker

# Or with Podman
./total-refresh.sh --new --container-tool podman

# Using SSL then run the above with --ssl
./total-refresh.sh --new --container-tool docker --ssl

Script Options

• --new: Sets up a new environment, building all containers from scratch.
• --container-tool <docker|podman>: Specifies the container runtime to use.
• --ssl: Enables SSL and manages certificates.
• --renew-certs: Renews SSL certificates without a full environment refresh.
• --clean-volumes: Removes all Docker volumes, deleting persistent data (e.g., database and search indexes). Use with caution.
• --no-pull: Skips pulling the latest code from Git repositories. This is useful for development when you want to test local changes without them being overwritten. Cannot be used with --new or --renew-certs.
• --verbose: Enables detailed logging for debugging.
• --help: Displays the help message.

Common Use Cases

Refresh existing environment (pulls latest code and rebuilds):

./total-refresh.sh --container-tool docker

Refresh with SSL certificate generation:

./total-refresh.sh --container-tool docker --ssl

Certificate renewal only (for cron jobs):

./total-refresh.sh --container-tool docker --renew-certs

Clean refresh (removes all volumes - destroys data!):

./total-refresh.sh --container-tool docker --clean-volumes

Run refresh without pulling code (for local development):

./total-refresh.sh --container-tool docker --no-pull

Script Features

• Automated Dependency Checking: Validates Docker/Podman, Git, and jq installation
• Environment Validation: Checks .env file configuration and required variables
• Submodule Management: Handles Git submodule initialization and updates with fallback strategies
• Service Orchestration: Manages proper startup sequence and health monitoring
• SSL Support: Optional Let's Encrypt certificate generation with Certbot
• Resource Cleanup: Intelligent cleanup of unused containers, images, and optionally volumes
• Health Monitoring: Waits for all services to reach healthy state (up to 10 minutes)
• Status Reporting: Displays service status and access URLs upon completion

Script Options

Option	Description
--new	Set up a new environment from scratch
--container-tool TOOL	Container tool to use (docker or podman) [Required]
--ssl	Enable SSL certificate generation with Certbot
--renew-certs	Renew SSL certificates only (no full refresh)
--clean-volumes	Remove all volumes during cleanup ⚠️ Destroys all data
--verbose	Enable verbose output for debugging
--help	Show detailed help information

Prerequisites for Script

• Container Runtime: Docker or Podman with Compose plugin

• Git: For repository and submodule management

• jq: For JSON parsing of container status

  # macOS
  brew install jq
  
  # Ubuntu/Debian
  sudo apt-get install jq
  
  
  # CentOS/RHEL
  sudo yum install jq

• Environment File: Valid .env file (copy from sample.env)

• SSL (Optional): Google Cloud credentials in certbot/gcp-key.json

Method 2: Manual Setup

For users who prefer manual control or are on Windows:

# Clone the repository with all submodules
git clone --recursive https://github.com/HGNC/pgnc-external-stack.git

# Enter the project directory
cd pgnc-external-stack

# Configure environment variables
cp sample.env .env
# Edit .env with your specific configuration

# Start all services
docker compose up -d

# Or using Podman
podman compose up -d

Accessing the Application

• Website: http://localhost:8080
• API Documentation: http://localhost:3000/api
• Solr Admin: http://localhost:8983/solr (Apache Solr 9.9.0)

SSL/HTTPS Setup

To enable SSL certificates for production deployment:

# Build the certbot container
docker compose build certbot

# Generate SSL certificates
docker compose run --rm certbot

SSL Certificate Auto-Renewal

Let's Encrypt certificates expire every 90 days. The PGNC stack includes automated renewal functionality to prevent service interruption:

# Manual certificate renewal
./total-refresh.sh --container-tool docker --renew-certs


# Or use the dedicated renewal script
./cert-renewal.sh docker

For automated renewal with cron:

# Edit crontab to run twice daily
crontab -e


# Add this line (adjust path to your project directory):
30 2,14 * * * cd /path/to/pgnc-external-stack && ./cert-renewal.sh docker >> /var/log/pgnc-cert-renewal.log 2>&1

For detailed setup instructions, see:

• 📋 SSL Renewal Setup Guide - Complete configuration instructions

Note: Ignore transaction-related output messages - these are normal operation logs, not errors.

Example of expected (non-error) output:

Hook '--manual-cleanup-hook' for plant.genenames.org ran with error output:
 Transaction started [transaction.yaml].
 Record removal appended to transaction at [transaction.yaml].
 Executed transaction [transaction.yaml] for managed-zone [genenames-org].

Development & Maintenance

Using the total-refresh.sh Script (Recommended)

The automated script handles most maintenance tasks:

# Standard refresh (updates code, rebuilds containers)
./total-refresh.sh --container-tool docker

# Refresh with SSL certificate renewal
./total-refresh.sh --container-tool docker --ssl

# Certificate renewal only (for cron jobs)
./total-refresh.sh --container-tool docker --renew-certs

# Deep clean refresh (removes all data volumes)
./total-refresh.sh --container-tool docker --clean-volumes


# Verbose output for troubleshooting
./total-refresh.sh --container-tool docker --verbose

What the Script Does

For New Environments (--new flag):

1. Validates system prerequisites (Docker/Podman, Git, jq)

2. Checks environment configuration (.env file)

3. Initializes Git submodules from scratch

4. Builds all container images with fresh cache

5. Starts services in proper dependency order

6. Monitors service health until all are ready

7. Optionally generates SSL certificates

8. Displays status and access URLs

For Environment Refresh (default):

1. Stops all running services gracefully

2. Cleans up unused containers, images, and networks

3. Optionally removes data volumes (with --clean-volumes)

4. Updates Git submodules with fallback strategies

5. Rebuilds all container images

6. Restarts services with health monitoring

7. Optionally renews SSL certificates

8. Reports final status

Service Health Monitoring

The script monitors different service types appropriately:

• Long-running services (database, API, frontend, Solr): Must reach "healthy" status
• Task services (Python data loader): Must exit with code 0
• Nginx: Health depends on SSL configuration

Timeout: 10 minutes with progress updates every 30 seconds.

Troubleshooting with the Script

# Check what the script requires
./total-refresh.sh --help

# Run with verbose output
./total-refresh.sh --container-tool docker --verbose

# If services fail to start, check logs

docker compose logs -f

# For submodule issues, the script provides manual commands
git submodule status

git submodule deinit --all -f
git submodule update --init --recursive

Script Error Handling

The script uses set -euo pipefail for strict error handling:

• Exit Code 0: Successful completion
• Exit Code 1: Error occurred (invalid arguments, missing dependencies, setup failure)

Common error scenarios and solutions:

• Missing container tool: Install Docker or Podman with Compose plugin

• Missing jq: Install jq for JSON parsing (brew install jq on macOS)

• Invalid .env: Copy sample.env to .env and configure all required variables

• Submodule failures: Script provides fallback strategies and manual recovery commands

• Service health timeouts: Check container logs for specific service errors

The script provides colored output:

• 🔵 Blue [INFO]: General information
• 🟢 Green [SUCCESS]: Successful operations
• 🟡 Yellow [WARNING]: Non-critical issues
• 🔴 Red [ERROR]: Critical failures

Manual Maintenance (Alternative)

For users who prefer manual control:

# Stop all services
docker compose down

# Clean up resources
docker image prune --all --force
docker volume prune --force  
docker network prune --force

# Update code
git pull --recurse-submodules

# Restart services
docker compose up -d

Working with Python Components

The python/ directory contains data processing utilities:

# Install Python dependencies
cd python
pip install -r requirements.txt

# Run data loading scripts
python bin/data-load/main.py


# Run data update scripts  
python bin/data-update/main.py

# Run tests
pytest tests/

See python/README.md for detailed Python component documentation.

Testing

The project includes comprehensive test suites:

Shell Tests

• Location: tests/
• What: Bash tests for total-refresh.sh and cert-renewal.sh
• Run (npm):

npm run test:shell        # run both suites
npm run test:shell:total  # total-refresh only
npm run test:shell:certs  # cert-renewal only

• Run (direct):

bash tests/test_total-refresh.sh
bash tests/test_cert-renewal.sh

Prerequisites: macOS/Linux shell, bash, and typical coreutils. Tests create temp dirs and mock dependencies; they do not modify your repo.

Python Tests

• Location: python/tests/
• Framework: pytest with comprehensive coverage
• Coverage: Gene models, data processing, API integrations
• Run: cd python && pytest tests/ -v

Frontend Tests

• Framework: Jest (replaces deprecated Karma)
• Location: angular/src/
• Run: cd angular && npm test

API Tests

• Framework: Jest with NestJS testing utilities
• Location: api/src/
• Run: cd api && npm test

For detailed testing information, see python/TESTING_SUMMARY.md.

Contributing

We welcome contributions! Please follow these steps:

1. Fork the repository

2. Create a feature branch: git checkout -b feature/amazing-feature

3. Commit your changes: git commit -m 'Add amazing feature'

4. Test your changes thoroughly

5. Push to your branch: git push origin feature/amazing-feature

6. Submit a Pull Request

Development Guidelines

• Follow existing code style and conventions
• Include tests for new functionality
• Update documentation as needed
• Ensure all tests pass before submitting PR

Troubleshooting

Common Issues

Port Conflicts: If you see port binding errors, check that ports 8080, 3000, 8983, 5432 are available.

Docker Issues: Try cleaning up Docker resources:

docker system prune -a
docker volume prune

Database Connection: Verify your .env file has correct database credentials.

SSL Certificate Issues: Ensure your GCP service account has proper DNS permissions.

Documentation

• Python Components
• Angular Frontend
• Testing Guide
• Pylance Configuration
• SSL Certificate Renewal Setup
• Changelog

Security

• Never commit secrets: Keep real values only in your local .env (confirmed in .gitignore). Use sample.env as the template with placeholders only.
• Where to store keys:
  • CLI: .env
  • VS Code MCP tools: .vscode/mcp.json env section (keys only)
  • All other config (models, params) lives in .taskmaster/config.json managed via task-master models.
• Rotation: If a key is ever exposed, rotate it immediately in the provider dashboard and replace locally. Consider invalidating sessions/tokens as required.
• Docs hygiene: Use placeholders in documentation; do not paste real tokens or JWTs into examples.

Technology Stack

• Frontend: Angular 19.1+, TypeScript, RxJS
• Backend: NestJS 10.x, Node.js, TypeScript
• Database: PostgreSQL 17.0
• Search: Apache Solr 9.9.0
• Containers: Docker/Podman with Docker Compose
• Web Server: Nginx (reverse proxy)
• SSL: Let's Encrypt with Certbot
• Data Processing: Python 3.13+
• Testing: Jest, pytest 8.4+
• Cloud: Google Cloud Platform (DNS management)

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).

• Commercial Use: Permitted with source disclosure
• Distribution: Must include license and copyright notice
• Patent Use: Expressly granted
• Private Use: Permitted
• Network Use: Must provide source code

See the LICENSE file for the complete terms.

---

Plant Gene Nomenclature Committee (PGNC)
Standardizing plant gene nomenclature for the global research community