Import PubChem Database

Import PubChem Database

PubChem is an open chemistry database at the National Institutes of Health (NIH).

We're going to import necessary data for Periodum in order to provide more detail about compounds etc.

Import Script

This script will be a part of the Periodum-API project. It will be manually executed since we need to run it a few times.

However, there's a rate limiter while using the PubChem's REST API. Therefore the script must follow the limits and behave accordingly.

The script's model would be like;

• Send a request here: https://pubchem.ncbi.nlm.nih.gov/rest/pug_view/data/compound/{compund_id}/JSON/ ex: You may use 222 as compound id
• This request will return a full record of the compound information in JSON format.
• Parse only necessary data objects for Periodum that provided here and save it to the DB with the compound_id combine with data property. DB schema will be defined later in the API project.

This is basic behaviour of the script. Since there're over 110M+ records the script must expect a start (Integer) and and end (Integer) compound id as parameters to iterate the requests, by following the rate limits. So, we can run it on multiple servers at the same time. This parameters must be passed through command args.

PubChem Rate Limiter

Standard Time Limit

PubChem has a standard time limit of 30 seconds per web service request. If a request is not completed within the 30-second limit for any reason, a timeout error will be returned. To work around certain slower operations, one may use an asynchronous approach, where a so-called key is returned as a response to the initial request. This key is then used to check periodically whether the operation has finished, and, when complete, retrieve the results.

Request Volume Limitations

• No more than 5 requests per second.
• No more than 400 requests per minute.
• No longer than 300 second running time per minute.

Additional Limits

• Dynamic Request Throttling
• MaxRecords - maximum number of hits: 2M