Feat/add orange cameroun MNO data simulator#36
Merged
Conversation
- Rename SFTP/DB users: cml_user/user1 -> demo_openmrg, user2 -> demo_orange_cameroun
- New DB migration 007 renames roles and relabels existing data rows
- database/init.sql updated for fresh deployments
- ssh_keys/.gitignore updated; user1/ -> demo_openmrg/, user2/ -> demo_orange_cameroun/
- New dedicated SSH key pair for demo_orange_cameroun (id_rsa_orange_cameroun)
- sftp_receiver entrypoint and integration test updated
- Add second MNO data source simulator for Orange Cameroun (cml_raw_douala.nc)
- New services: mno_simulator_orange_cameroun, archive_generator_orange_cameroun,
archive_loader_orange_cameroun, parser_orange_cameroun
- All existing services renamed from generic to *_openmrg suffix
- 31-day archive at 15-min native resolution (ARCHIVE_INTERVAL_SECONDS=900)
- Support rsl_min/tsl_min variable names (Phase 1: use min values as rsl/tsl)
- data_generator.py: NETCDF_RSL_VAR / NETCDF_TSL_VAR env vars
- generate_archive.py: uses generator.rsl_var / tsl_var
- parse_netcdf_archive.py: ARCHIVE_RSL_VAR / ARCHIVE_TSL_VAR / ARCHIVE_USER_ID
- Filter fully-NaN sublinks at metadata extraction time
- Only sublinks with >=1 valid value are written to DB and CSV
- Handles both (sublink_id, cml_id, time) and (cml_id, sublink_id, time) dim orders
- Handles datasets without polarization coordinate (stores NULL)
- Handles pre-computed length coordinate (skips haversine calculation)
- database/init_archive_data.sh: ARCHIVE_USER_ID tags all loaded rows correctly
Phase 2 (next PR): extend DB schema + parser + Grafana for min/avg/max columns
Compressed TimescaleDB hypertable chunks cannot be updated without first decompressing them, which on months of raw data is prohibitively slow. cml_metadata and cml_stats (small, uncompressed) are still relabelled. cml_data rows retain user_id='user1' for historical data; new rows from the renamed parser will be written as 'demo_openmrg' going forward. Grafana (connecting as myuser/superuser) sees all rows regardless. Instructions for a full relabel when needed are left as comments in the migration file.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #36 +/- ##
==========================================
+ Coverage 77.76% 78.93% +1.17%
==========================================
Files 24 24
Lines 2190 2317 +127
==========================================
+ Hits 1703 1829 +126
- Misses 487 488 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…rray mocks for RSL and TSL
…i-user setup - Add SFTP connection env var overrides in mno_data_source_simulator/main.py so SFTP_USERNAME and other env vars from docker-compose override config.yml (fixes auth failure: was connecting as cml_user instead of demo_openmrg) - Update scripts/run_e2e_test_locally.sh to use correct service names (parser_openmrg, mno_simulator_openmrg), SFTP path (/home/demo_openmrg/uploads), SSH key dirs (demo_openmrg/demo_orange_cameroun), and set OPENMRG_NETCDF_FILE env var to avoid downloading the large 3-month dataset during local testing - Fix SFTP directory path in CI workflow (cml_user -> demo_openmrg)
cchwala
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Orange Cameroun second MNO simulator
Summary
Adds a second MNO data source simulator representing Orange Cameroun, alongside the existing OpenMRG simulator. Renames the original SFTP user from
cml_user/user1todemo_openmrgfor consistency, and introducesdemo_orange_camerounas the second user. Both users now have DB-enforced row-level separation.Changes
New MNO simulator
mno_simulator_orange_camerounservice todocker-compose.ymlwith its own SFTP credentials, SSH key pair, archive generator, and archive loadercml_raw_douala.ncNetCDF source file for the Orange Cameroun dataset (Douala, Cameroon)data_generator.pyandparse_netcdf_archive.pyto support configurable RSL/TSL variable names (rsl_min,tsl_min) needed by the Douala datasetUser rename
cml_user/user1→demo_openmrg;user2→demo_orange_camerounacross SFTP config, SSH keys, DB roles, and all services007_rename_users_add_orange_cameroun.sqlhandles in-place rename of existing data rows and rolesInfrastructure
mno_data_source_simulator/main.py: env var overrides for SFTP config are now applied before the SFTP client is initialised (bug fix — previously overrides had no effect)parser_orange_camerounservice added, sharing the same parser image withUSER_ID=demo_orange_camerounCI / testing
test_parse_netcdf_archive.pyextended to cover the new RSL/TSL variable name configurationTesting