Greenplum parallel import and export OSS data plugin oss_ext
Greenplum supports parallel import from OSS or export to OSS through external tables (which is called the gpossext function). It can also compress OSS external table files in gzip format to reduce the storage space and the costs.
The gpossext function can read or write text/csv files or text/csv files in gzip format.
2.what is oss
Alibaba Cloud Object Storage Service (OSS) is a storage service that enables you to store, back up, and archive any amount of data in the cloud. OSS is a cost-effective, highly secure, and highly reliable cloud storage solution. It uses RESTful APIs and is designed for 99.999999999% (11 nines) durability and 99.99% availability. Using OSS, you can store and retrieve any type of data at any time, from anywhere on the web.
You can use API and SDK interfaces provided by Alibaba Cloud or OSS migration tools to transfer massive amounts of data into or out of Alibaba Cloud OSS. You can use the Standard storage class of OSS to store image, audio, and video files for apps and large websites. You can use the Infrequent Access (IA) or Archive storage class as a low-cost solution for backup and archiving of infrequently accessed data.
1. oss c sdk
the stable version of osslib already included in the code
apr apr-devel apr-util apr-util-devel mxml mxml-devel
Pigz is installed into the Greenplum bin directory to compress and write data from Greenplum to oss
4. testcase dependency
osscmd object commands
The performance of oss_ext read and write oss increases with the increase of Greenplum compute nodes. It supports asynchronous reading of data in oss and parallel compression of data to write oss.
The oss has a traffic limit of about 5Gbyte/s. If there is a demand, you can request bandwidth from the oss product.