|
| 1 | +# cnpmcore - Private NPM Registry for Enterprise |
| 2 | + |
| 3 | +cnpmcore is a TypeScript-based private NPM registry implementation built with Egg.js framework. It provides enterprise-grade package management with support for MySQL/PostgreSQL databases, Redis caching, and optional Elasticsearch. |
| 4 | + |
| 5 | +**ALWAYS reference these instructions first** and fallback to search or bash commands only when you encounter unexpected information that does not match the information here. |
| 6 | + |
| 7 | +## Prerequisites and Environment Setup |
| 8 | + |
| 9 | +- **Node.js**: Version 20.18.0 or higher (required by engines field in package.json) |
| 10 | +- **Database**: MySQL 5.7+ or PostgreSQL 17+ |
| 11 | +- **Cache**: Redis 6+ |
| 12 | +- **Optional**: Elasticsearch 8.x for enhanced search capabilities |
| 13 | + |
| 14 | +## Working Effectively |
| 15 | + |
| 16 | +### Bootstrap and Build |
| 17 | +```bash |
| 18 | +# Install dependencies (takes ~2 minutes) |
| 19 | +npm install |
| 20 | + |
| 21 | +# Copy environment configuration |
| 22 | +cp .env.example .env |
| 23 | + |
| 24 | +# Lint code (very fast, <1 second) |
| 25 | +npm run lint |
| 26 | + |
| 27 | +# Fix linting issues |
| 28 | +npm run lint:fix |
| 29 | + |
| 30 | +# Build TypeScript (takes ~6 seconds) |
| 31 | +npm run tsc |
| 32 | + |
| 33 | +# Production build (takes ~6 seconds) |
| 34 | +npm run tsc:prod |
| 35 | +``` |
| 36 | + |
| 37 | +### Database Setup - MySQL (Recommended for Development) |
| 38 | +```bash |
| 39 | +# Start MySQL + Redis services via Docker (takes ~1 minute to pull images initially) |
| 40 | +docker compose -f docker-compose.yml up -d |
| 41 | + |
| 42 | +# Verify services are running |
| 43 | +docker compose ps |
| 44 | + |
| 45 | +# Initialize database (takes <2 seconds) |
| 46 | +CNPMCORE_DATABASE_NAME=cnpmcore bash ./prepare-database-mysql.sh |
| 47 | + |
| 48 | +# For tests, create test database |
| 49 | +mysql -h 127.0.0.1 -P 3306 -u root -e "CREATE DATABASE cnpmcore_unittest;" |
| 50 | +``` |
| 51 | + |
| 52 | +### Database Setup - PostgreSQL (Alternative) |
| 53 | +```bash |
| 54 | +# Start PostgreSQL + Redis services via Docker |
| 55 | +docker compose -f docker-compose-postgres.yml up -d |
| 56 | + |
| 57 | +# Initialize database (takes <1 second) |
| 58 | +CNPMCORE_DATABASE_NAME=cnpmcore bash ./prepare-database-postgresql.sh |
| 59 | +``` |
| 60 | + |
| 61 | +### Development Server |
| 62 | +```bash |
| 63 | +# MySQL development server (starts in ~20 seconds) |
| 64 | +npm run dev |
| 65 | +# Server runs on http://127.0.0.1:7001 |
| 66 | + |
| 67 | +# PostgreSQL development server |
| 68 | +npm run dev:postgresql |
| 69 | +# Server runs on http://127.0.0.1:7001 |
| 70 | +``` |
| 71 | + |
| 72 | +### Testing |
| 73 | +```bash |
| 74 | +# Run full test suite with MySQL - NEVER CANCEL: Takes 4+ minutes. Set timeout to 10+ minutes. |
| 75 | +npm run test |
| 76 | + |
| 77 | +# Run full test suite with PostgreSQL - NEVER CANCEL: Takes 4+ minutes. Set timeout to 10+ minutes. |
| 78 | +npm run test:postgresql |
| 79 | + |
| 80 | +# Run single test file (for faster iteration, takes ~12 seconds) |
| 81 | +npm run test:local test/common/CryptoUtil.test.ts |
| 82 | + |
| 83 | +# Test coverage with MySQL - NEVER CANCEL: Takes 5+ minutes. Set timeout to 15+ minutes. |
| 84 | +npm run cov |
| 85 | + |
| 86 | +# Test coverage with PostgreSQL - NEVER CANCEL: Takes 5+ minutes. Set timeout to 15+ minutes. |
| 87 | +npm run cov:postgresql |
| 88 | +``` |
| 89 | + |
| 90 | +**CRITICAL TESTING NOTES:** |
| 91 | +- **NEVER CANCEL** build or test commands - they may take 4-15 minutes to complete |
| 92 | +- Individual test files run much faster (~12 seconds) for development iteration |
| 93 | +- Full test suite processes 100+ test files and requires database initialization |
| 94 | +- Test failures may occur in CI environment; use individual test files for validation |
| 95 | + |
| 96 | +### Production Commands |
| 97 | +```bash |
| 98 | +# CI pipeline commands - NEVER CANCEL: Takes 5+ minutes. Set timeout to 15+ minutes. |
| 99 | +npm run ci # MySQL CI (includes lint, test, coverage, build) |
| 100 | +npm run ci:postgresql # PostgreSQL CI |
| 101 | + |
| 102 | +# Production start/stop |
| 103 | +npm run start # Start as daemon |
| 104 | +npm run stop # Stop daemon |
| 105 | +npm run start:foreground # Start in foreground for debugging |
| 106 | +``` |
| 107 | + |
| 108 | +## Validation Scenarios |
| 109 | + |
| 110 | +**ALWAYS manually validate changes** by running through these scenarios: |
| 111 | + |
| 112 | +### Basic API Validation |
| 113 | +```bash |
| 114 | +# Start development server |
| 115 | +npm run dev |
| 116 | + |
| 117 | +# Test registry root endpoint |
| 118 | +curl http://127.0.0.1:7001 |
| 119 | +# Should return JSON with app metadata and stats |
| 120 | + |
| 121 | +# Test authentication endpoint |
| 122 | +curl http://127.0.0.1:7001/-/whoami |
| 123 | +# Should return authentication error (expected when not logged in) |
| 124 | + |
| 125 | +# Test package listing (initially empty) |
| 126 | +curl http://127.0.0.1:7001/-/all |
| 127 | +``` |
| 128 | + |
| 129 | +### Admin User Setup and Package Publishing |
| 130 | +```bash |
| 131 | +# Register admin user (cnpmcore_admin) - requires allowPublicRegistration=true in config |
| 132 | +npm login --registry=http://127.0.0.1:7001 |
| 133 | + |
| 134 | +# Verify login |
| 135 | +npm whoami --registry=http://127.0.0.1:7001 |
| 136 | + |
| 137 | +# Test package publishing |
| 138 | +npm publish --registry=http://127.0.0.1:7001 |
| 139 | +``` |
| 140 | + |
| 141 | +## Architecture and Navigation |
| 142 | + |
| 143 | +### Project Structure |
| 144 | +``` |
| 145 | +app/ |
| 146 | +├── common/ # Global utilities and adapters |
| 147 | +│ ├── adapter/ # External service adapters (NpmRegistry, Binary, etc.) |
| 148 | +│ └── enum/ # Shared enumerations |
| 149 | +├── core/ # Business logic layer |
| 150 | +│ ├── entity/ # Core domain models |
| 151 | +│ ├── event/ # Event handlers and async processing |
| 152 | +│ ├── service/ # Core business services |
| 153 | +│ └── util/ # Internal utilities |
| 154 | +├── port/ # Interface layer |
| 155 | +│ ├── controller/ # HTTP controllers |
| 156 | +│ ├── middleware/ # Express middleware |
| 157 | +│ ├── schedule/ # Background job schedulers |
| 158 | +│ └── webauth/ # WebAuth integration |
| 159 | +├── repository/ # Data access layer |
| 160 | +│ ├── model/ # ORM models |
| 161 | +│ └── util/ # Repository utilities |
| 162 | +└── infra/ # Infrastructure adapters |
| 163 | +``` |
| 164 | + |
| 165 | +### Key Services and Controllers |
| 166 | +- **PackageController**: Main package CRUD operations |
| 167 | +- **PackageManagerService**: Core package management business logic |
| 168 | +- **BinarySyncerService**: Binary package synchronization |
| 169 | +- **ChangesStreamService**: NPM registry change stream processing |
| 170 | +- **UserController**: User authentication and profile management |
| 171 | + |
| 172 | +### Configuration Files |
| 173 | +- `config/config.default.ts`: Main application configuration |
| 174 | +- `config/database.ts`: Database connection settings |
| 175 | +- `.env`: Environment-specific variables |
| 176 | +- `tsconfig.json`: TypeScript compilation settings |
| 177 | + |
| 178 | +## Common Development Tasks |
| 179 | + |
| 180 | +### Adding New Features |
| 181 | +1. **ALWAYS run** `npm run lint:fix` before making changes |
| 182 | +2. Add entity classes in `app/core/entity/` for new domain models |
| 183 | +3. Add services in `app/core/service/` for business logic |
| 184 | +4. Add controllers in `app/port/controller/` for HTTP endpoints |
| 185 | +5. Add repositories in `app/repository/` for data access |
| 186 | +6. **ALWAYS run** tests and linting before committing |
| 187 | + |
| 188 | +### Database Migrations |
| 189 | +- SQL files are in `sql/mysql/` and `sql/postgresql/` |
| 190 | +- Migration scripts automatically run during database preparation |
| 191 | +- **NEVER** modify existing migration files - only add new ones |
| 192 | + |
| 193 | +### Background Jobs |
| 194 | +- Schedulers are in `app/port/schedule/` |
| 195 | +- Include sync workers, cleanup tasks, and stream processors |
| 196 | +- Jobs run automatically when development server starts |
| 197 | + |
| 198 | +## Troubleshooting |
| 199 | + |
| 200 | +### Database Connection Issues |
| 201 | +```bash |
| 202 | +# Check if services are running |
| 203 | +docker compose ps |
| 204 | + |
| 205 | +# Reset MySQL environment |
| 206 | +docker compose -f docker-compose.yml down |
| 207 | +docker compose -f docker-compose.yml up -d |
| 208 | +CNPMCORE_DATABASE_NAME=cnpmcore bash ./prepare-database-mysql.sh |
| 209 | + |
| 210 | +# Reset PostgreSQL environment |
| 211 | +docker compose -f docker-compose-postgres.yml down |
| 212 | +docker compose -f docker-compose-postgres.yml up -d |
| 213 | +CNPMCORE_DATABASE_NAME=cnpmcore bash ./prepare-database-postgresql.sh |
| 214 | +``` |
| 215 | + |
| 216 | +### Build Issues |
| 217 | +```bash |
| 218 | +# Clean and rebuild |
| 219 | +npm run clean |
| 220 | +npm run tsc |
| 221 | + |
| 222 | +# Check TypeScript configuration |
| 223 | +npx tsc --noEmit |
| 224 | +``` |
| 225 | + |
| 226 | +### Test Issues |
| 227 | +```bash |
| 228 | +# Create missing test database |
| 229 | +mysql -h 127.0.0.1 -P 3306 -u root -e "CREATE DATABASE cnpmcore_unittest;" |
| 230 | + |
| 231 | +# Run single test for debugging |
| 232 | +npm run test:local test/common/CryptoUtil.test.ts |
| 233 | +``` |
| 234 | + |
| 235 | +## CI/CD Integration |
| 236 | + |
| 237 | +The project uses GitHub Actions with workflows in `.github/workflows/`: |
| 238 | +- `nodejs.yml`: Main CI pipeline with MySQL, PostgreSQL, and Elasticsearch testing |
| 239 | +- Multiple Node.js versions tested: 20, 22, 24 |
| 240 | +- **CRITICAL**: CI jobs include long-running tests that can take 15+ minutes per database type |
| 241 | + |
| 242 | +### Pre-commit Validation |
| 243 | +**ALWAYS run before committing:** |
| 244 | +```bash |
| 245 | +npm run lint:fix # Fix linting issues |
| 246 | +npm run tsc # Verify TypeScript compilation |
| 247 | +npm run test:local test/path/to/relevant.test.ts # Run relevant tests |
| 248 | +``` |
| 249 | + |
| 250 | +## Docker Support |
| 251 | + |
| 252 | +### Development Environments |
| 253 | +- `docker-compose.yml`: MySQL + Redis + phpMyAdmin |
| 254 | +- `docker-compose-postgres.yml`: PostgreSQL + Redis + pgAdmin |
| 255 | +- `docker-compose-es.yml`: Elasticsearch integration |
| 256 | + |
| 257 | +### Production Images |
| 258 | +```bash |
| 259 | +# Build Alpine image |
| 260 | +npm run images:alpine |
| 261 | + |
| 262 | +# Build Debian image |
| 263 | +npm run images:debian |
| 264 | +``` |
| 265 | + |
| 266 | +## External Dependencies |
| 267 | + |
| 268 | +- **Database**: MySQL 9.x or PostgreSQL 17+ |
| 269 | +- **Cache**: Redis 6+ |
| 270 | +- **Search**: Elasticsearch 8.x (optional) |
| 271 | +- **Storage**: Local filesystem or S3-compatible storage |
| 272 | +- **Framework**: Egg.js with extensive TypeScript integration |
| 273 | + |
| 274 | +## Performance Notes |
| 275 | + |
| 276 | +- **Startup Time**: ~20 seconds for development server |
| 277 | +- **Build Time**: ~6 seconds for TypeScript compilation |
| 278 | +- **Test Time**: 4-15 minutes for full suite (database dependent) |
| 279 | +- **Package Installation**: ~2 minutes for npm install |
| 280 | +- **Database Init**: <2 seconds for either MySQL or PostgreSQL |
| 281 | + |
| 282 | +Always account for these timings when setting timeouts for automated processes. |
0 commit comments