I learned about OneMap through my work, as we needed to integrate its Map Service into our projects. During this process, I discovered the Population Query section, which offered a wealth of information. This sparked my interest to collect this data and store it in my S3 Bucket for further analysis and use.
Here is my S3 Bucket onemap-s3 with all the categories of data as my file structure. As the Data is not continuously updated, there was no reason to setup a AWS Lambda Function to automate this data collection process.
There are two ways to perform API calls:
- Synchronous Requests (slow)
- Asyncronous Requests (fast)
Synchronous requests are processed sequentially, meaning each request is sent and must be fully completed before the next request can begin this can lead to delays. However the positives of using Synchronus requests is that this method ensures each task is finished in order.
Asyncronous requests are processed in parallel, without waiting for one to complete before starting the next. This leads to overall faster speeds when collecting api data.
As we can see the data collected on the left (Syncronous) processes them one-by-one. The data collected on the right (Asyncronous) is faster and in parallel.