CSV export from SQL server DB using Connector X #374
Unanswered
vishalkhondre
asked this question in
Q&A
Replies: 1 comment
-
Hi @vishalkhondre , I think it depends on the bottleneck of the process. (which part is more time costly: If import connectorx as cx
postgres_url = "postgresql://username:password@server:port/database"
queries = ["SELECT * FROM lineitem WHERE l_orderkey <= 30000000", "SELECT * FROM lineitem WHERE l_orderkey > 30000000"]
cx.read_sql(postgres_url, queries) In this case the two queries will be run in parallel and speed up the process. If |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi Team,
I'm using below code to export data into CSV.
I'm trying export ~4 M records and its taking ~25 to 30 minutes.
Is there a way that I can provide batchsize in below code to speedup the export in CSV?
Also unfortunately there is no any column that I can use for partition. I believe for partition we need integer datatype column.
df = cx.read_sql(url, tabledetails.query)
df.to_csv(output_file, mode=mode, header=header, index=False)
Beta Was this translation helpful? Give feedback.
All reactions