Home

Parallel.GAMIT (PG) Wiki

Parallel.GAMIT is a python wrapper to process GPS/GNSS data using PPP and GAMIT/GLOBK. This wiki page provides the basic installation and execution guide and how to get started with the basic PG commands. PG depends on multiple packages and applications, some of them not easy to install. Setting up a cluster is not the easiest task, but hopefully this Wiki will get you started.

PG dependencies

PG depends of the following packages and applications (all of them are open source or you can get them by sending an email to the authors):

1. GAMIT/GLOBK and its dependencies: this is one of the most popular GNSS processing tools and can be obtained by contacting MIT.
2. NRCan-CGS GPSPACE and its dependencies: FORTRAN program to obtain GNSS Precise Point Positioning (PPP) solutions. We recommend this branch: https://github.com/pomath/GPSPACE
3. Postgres database engine: can be downloaded from the official Postgres site.
4. The following applications/programs: GFZRNX (https://dataservices.gfz-potsdam.de/panmetaworks/showshort.php?id=escidoc:1577894), crx2rnx (Hatanaka), rnx2crx.
5. Python version 3 or above.
6. Module dependencies are listed in INSTALL.md

On of these modules, dispy, is used to run the cluster and handle its jobs and requires some extra work that will be explained later. We will not go into detail about setting up the environment and compiling/installing the modules and packages needed to work with PG, since most of them already have their own instructions.

The structure of PG

PG is structured in the following way:

1. Folder classes: where all the shared objects and libraries live. No command is executed from this folder.
2. Folder parallel_gamit: objects and execution script (pyParallelGamit) to run PG
3. Folder com: Command scripts used to import data, change stations metadata, run PPP, etc.
4. Folder stacker: Location of polyhedron stacker scripts and objects.
5. Folder scripts and database: Location of generic scripts used to run PG and setup cluster, database, etc.

To run

Getting started:

1. Make sure that python 3 is activated!
2. Mirror the working directory onto all of your desired nodes and make sure that the folder where the command is run from is accessible! Specifically you need to have permissions to create the production folder. This folder is created when the dispy server program is run. This execution has to be done locally in the executing node.
3. Make sure that the shared drive is mounted!
   • mkdir /tmp/UsersDrive/
   • mount -t afp "afp://[username]:[password]@[hostname]/[diskname]" /tmp/UsersDrive/
   • Unmount with umount /tmp/UsersDrive

Some definitions:

• Archive: set of standard unix file structure directories where the RINEX files live. These are accessible to unix.
• Database: is the set of PostgreSQL files that are protected, and generally not understandable or available directly from unix, you have to go through the database engine.
• Gatekeeper: Python program that is used to "link" the archive and the database. Nothing goes into the archive without creating a record in the database. The gatekeeper consists of the following parts:
  • com/ArchiveService.py
  • com/ScanArchive.py
  • Each one is described in the program flow section.
• Metadata: the information associated with the station (station_info) but not the daily coordinate itself.

Program flow

com/ArchiveService.py

This program is responsible for scanning the repository for new RINEX files. It tries to match each file to a particular network-station pair using the coordinate obtained from the file (either using PPP or pseudorange positioning). The logic works as follows:

• When a matching network-station is found and no RINEX file exists for the day, the RINEX file is immediately moved into the Archive and a record is created in the database.
• If a RINEX file exists for the day and the incoming file is identical (size and length) to the one in the database, then the file is discarded and an event is logged in the events table. If the file is not identical, then the file is moved into the Archive and a record is added in table "rinex". A different process will later determine which file should be promoted as the "principal" RINEX file.
• If more than one station is found (No match, but candidates), an event is logged and the file is moved to the retry folder.
• When no match is found, the Gatekeeper assumes it's a new station and a lock is created in the table "locks" so that it doesn't get processed again by ArchiveService.py until the lock is removed. All files belonging to this new station (that fall within 100 meters of one another) will be grouped in a new station entry in the "stations" table with a temporary network code "???". This will be reflected by the "locks" table, where each file is associated with a NetworkCode and StationCode. If a RINEX file produces a faulty coordinate (that falls outside the 100 m range from the rest of the files), the Gatekeeper will assume that this file belongs to a different station that has the same name. It will therefore create another entry in the "stations" table using the same StationCode but a different NetworkCode (usually ??0, ??1, ??2, etc) since the "stations" table has a primary key that prevents the insertion of the same NetworkCode-StationCode pair. The file does NOT get moved out of the data_in folder.

Procedure for execution:

1. Add your rinex files to directory /repository/data_in/
2. Run com/ArchiveService.py. This process does not require any metadata (station info).
3. Inspect the events table, the retry folder and the error log to determine any possible problems during execution.
4. If new stations are found, there will be entries with "NetworkCode" = ??? in the stations table. These stations should be assigned to a network that does not have a StationCode conflict by assigning the appropriate NetworkCode field. If by accident you assign a NetworkCode where that StationCode already exists, the database will produce an error and the NetworkCode will not be assigned. The NetworkCode has to exist before assigning it. If it doesn't exists, add it with the following SQL command:
   • INSERT INTO networks ("NetworkCode", "NetworkName") VALUES ('sog', 'South Georgia Network');
   • Network codes are three lower case letters, Network Names are human readable names.
5. After assigning a NetworkCode, the database will automatically "remove" the lock for the files in the data_in and these will get processed during the next execution of pyArchiveService.py.
6. It is highly recommended to add the Station Information metadata immediately after assigning a NetworkCode to a station. This is just to keep consistency. No PPP coordinates will be added until the metadata is complete. The metadata can be added using the pyScanArchive interface, which will be discussed later.

com/ScanArchive.py

This program performs most of the "housekeeping" processes for the Gatekeeper. It's not meant to run as a service looking for RINEX to add to the database, although you need to invoke it whenever new data comes into the Archive. The tasks that pyScanArchive performs are the following:

• Produces PPP coordinates (with APR quality) using the RINEX files in the rinex table of the database (--ppp option).
• It can scan a pre-existing Archive directory structure to incorporate the RINEX files into the database. This is usually the "starter" for a new database based on existing RINEX data (--rinex option).
• Can calculate the OTL parameters and approximate coordinates of a station that has been recently added to the database using the --rinex option (--otl option)
• It incorporates station information files by either scanning an existing archive structure or using a single station.info file. It can also read the standard input from UNIX to get data from other tools such as the UNAVCO web service of simply the "cat" command in UNIX. See the help for the --stninfo option.
• Resolves the RINEX file conflicts by determining which file should be the "primary" file for a station and which one should stay in rinex_extra. This is a slow process because it cannot run in parallel!

There are now a number of cases:

• If this crude PPP location (at this point the receiver, antenna, etc,., metadata was not included in the PPP processing, so it is not a precise location) is within 100 m of some station in the database, then the station_code of the candidate rinex file is checked against the station_code of the very nearby station. If the two station_codes do not agree, the candidate rinex file is flagged and ... . If they agree the candidate rinex file is identified as belonging to that network/station_code.
• The candidate rinex file is now associated with a known network/station. If there is an identical rinex file already in the archive, then the candidate rinex file is a duplicate and pyArchiveService.py will delete it from the repository.
• If there is a rinex file with the same name (standard file names NNNNDDDZ.YYd.Z where NNNN is the 4 letter station code, DDD is the day of the year, Z is a sequence number [mostly used for very large high rate files], YY is a 2 digit year, d is for Hatanaka compressed, and Z is for unix compressed), but different size, already in the archive then the candidate rinex file is placed in rinex_extra (another routine/pass will sort out which is the principal rinex file later, usually it will be the biggest).
• If there is no rinex file with the same network/station_code already in the archive, pyArchiveService.py will move the candidate rinex file from the repository to the appropriate place in the archive. NOT QUITE
• If the approximate location, and 4 character station name/code, of the candidate rinex file is not associated with an existing network/station name/code in the database then the candidate rinex file will be associated with the ??? network, will have locks placed on it in the table in the database (the “lock” is in the database only, it keeps pyArchiveSerice.py, etc. from touching it. It does not do anything to the unix side of the file - you can still mess things up from unix). The candidate rinex file will be moved to data_retry/ and it will be ???.

Removing locks:

1. Check if the desired network name of the candidate rinex file is in the networks table.
2. If it is not in the networks table, then add the new network name it using:
   • INSERT INTO networks ("NetworkCode", "NetworkName") VALUES ('sog','South Georgia Network');
   • Network codes are three lower case letters, Network Names are human readable names.
3. Next change the network code of the station associated with the network ??? in the stations table
   • UPDATE stations SET "NetworkCode"='sog' WHERE ("StationCode",”NetworkCode”)=('sog1',’???’);
4. Once the new network code is set the station.info metadata needs to be updated (this is to obtain the necessary metadata to get precise PPP results for the precise apr location information in the database).
   • python pyScanArchive.py --stninfo --stninfo_path $PATHTOSTNINFO/station.info.sopac --stn SOG1 --net sog
5. Now, on future runs of pyArchiveService any rinex files placed in repository/data_in/ will be moved into the corresponding archive folder.

Now the program is ready to get some results, run pyScanArchive.py --ppp. On input at this step if it has the OTL and station info in the database

• It checks and runs PPP on EVERY entry in the rinex table in the database
• Even if you just need to run PPP for one new file, it will process/check all of them (with 25690 entries in the rinex table it takes about 21 minutes (5849.7790s) with 8 workers).
• The PPP output at this point is a high quality location using the equipment metadata (receiver, firmware, antenna, antenna response, etc.), antenna height/offset, and OTL. This high quality location is then saved in the database.

Loading new DB (outside of SQL):

• creatdb -d [newdbname]
• psql -d [newdbname] -f [pathtopsqldump]

Setting up the postgreSQL server:

1. Once the new database has been loaded restart it using:

   • pg_ctl -D /usr/local/var/postgres/ restart

2. Edit the file pg_hba.conf and add the following two lines in order to allow outside connections:

   • host all all 0.0.0.0/0 md5

3. Then open the file postgresql.conf and add the following line to listen for new connections:

   • listen_addresses = '*'

4. Once both files have been modified restart the server again.

PSQL Quirks:

1. You can start the server using pg_ctl -D /usr/local/var/postgres/ start, this starts up the server in the background so even if you close the terminal window it should still be running.
2. If you try to shutdown the server but still see the process running when you use ps -u [user] | grep sql then you have to shut it down using the aliased homebrew command ‘pg-stop’.
3. It’s best to dedicate a terminal window to the server startup so you can check it if there are any problems.

Parallel Python Cluster:

1. With pp installed on the master computer make sure that ppservers is set to *:port within the program where port can be whatever you want (default is 60000).
2. Then on the nodes run the script which comes with pp called ppserver.py with the flags -a and -d.
   • ppserver.py -a -d
   • -a for auto discovery, -d for debugging
3. The directory structure on the node should mirror that of the master’s and all data must be accessible to all nodes.

Fixing Outliers in Other Networks:

1. If you accidentally added a station in a new network and it has less than a month’s worth of data then follow these steps.
2. First check the database for stations with a short amount of data using:
   • select ("StationCode","DateEnd"-"DateStart") from stations order by "DateEnd"-"DateStart" asc;
3. Stations with a difference less than 0.068 have less than 24 days worth of data.
4. Check whether those stations already exist in other networks
   • select ("NetworkCode","StationCode","DateStart","DateEnd") from stations where "StationCode"='naus';
5. Look for the outlier in the rinex table (note the DOY and year) and see if it’s in the destination network already.
   • select * from rinex where ("NetworkCode","StationCode")=('bra',’naus’);
   • select * from rinex where ("NetworkCode","StationCode",”ObservationYear”,"ObservationDOY")=('sir','naus',’2016’,'326');
6. If it doesn’t already exist in the destination network then move it.
   • python pyIntegrityCheck.py --stn bra.naus --rename sir.naus
7. Once complete there shouldn’t be a rinex listed in the rinex table for the original network and station name.
8. Now the outlier can safely be deleted from the database.
   • delete from stationinfo where ("NetworkCode","StationCode")=('bra',’naus’);
   • delete from stations where ("NetworkCode","StationCode")=('bra','naus');
9. Check that the station info is still valid.
   • python pyIntegrityCheck.py --stn sir.naus --stninfo
10. This process must be completed for each station indiviually as of now, continue through the list created in step 2 until all stations have been fixed.

Useful psql Commands

1. Check which stations are missing station info: select "StationCode" from stations except select "StationCode" from stationinfo;