Fetch tables metadata takes too long #145
Comments
|
I see |
I came here today to say exactly this. I have 20-something schemas. Each with about 50 tables. The metadata fetch takes over an hour. the actual db dumping part takes probably 10 min to a local disk. TY for the tool, it does the thing, I just wish it were quicker. |
|
Hello! I think that it is a some kind of bug, we will check this. |
|
Let me know how I can help
…On Thu, Feb 3, 2022, 10:55 AM Evgeniy Rashchepkin ***@***.***> wrote:
Hello! I think that it is some kind of bug, we will check this.
—
Reply to this email directly, view it on GitHub
<#145 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACWRLSHM2YM422LQT53SIGTUZKQO7ANCNFSM5NNBFGCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
I have some ideas, could you please provide the database schema (the dump without data)? |
|
angryjohnnie <https://github.com/angryjohnnie>, are you able to easily?
this is a production database, I can't just share our schema, sorry.
…On Fri, Feb 4, 2022 at 8:09 AM Evgeniy Rashchepkin ***@***.***> wrote:
I have some ideas, could you please provide the database schema (the dump
without data)?
—
Reply to this email directly, view it on GitHub
<#145 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACWRLSBZSADUPV35GKTIRJTUZPFXNANCNFSM5NNBFGCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
im not sure what our policy is regarding sharing that schema. I can see if I can find out though. |
|
It looks like I am not allowed to share our schema either, unfortunately. |
|
OK, then we will try to fix this without these schemas. |
|
if you have branch or changes you would like to test, I can run it and
supply logs
…On Mon, Feb 7, 2022 at 11:50 AM Evgeniy Rashchepkin < ***@***.***> wrote:
OK, then we will try to fix this without these schemas.
—
Reply to this email directly, view it on GitHub
<#145 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACWRLSCGX7ZZZHVLPOSMG2TUZ7Z3TANCNFSM5NNBFGCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
I have a similar kind of issue, I have 4 schemas and around 20 tables in each of the schema. I also tried passing filter args using the cli <PG_DUMP_ARGS>. But it still takes more than 20 mins to fetch the metadata. |
|
Can anybody run dumping with the tracing branch and copy the output here? I can build a binary if required. |
|
I can if you can build a binary.
…On Sun, Feb 13, 2022 at 9:53 AM Evgeniy Rashchepkin < ***@***.***> wrote:
Can anybody run dumping with the tracing
<#153> branch and copy the
output here? I can build a binary if required.
—
Reply to this email directly, view it on GitHub
<#145 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACWRLSGW6MYEV3CG7BXNPWDU27AXJANCNFSM5NNBFGCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Platform? (win/linux/mac - Darwin or ARM) |
|
sorry, linux amd64
…On Tue, Feb 15, 2022 at 8:58 AM Evgeniy Rashchepkin < ***@***.***> wrote:
Platform? (win/linux/mac - Darwin or ARM)
—
Reply to this email directly, view it on GitHub
<#145 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACWRLSAQ4N7TRXUNFPKVYMTU3JLX7ANCNFSM5NNBFGCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
pg_datanymizer_debug_build --config ../config.yaml [...]
Inspecting tables ...
Inspecting completed in 4 seconds
Inspecting table dependencies...
and it has been stuck at the stage for hour+
…On Tue, Feb 15, 2022 at 9:52 AM Evgeniy Rashchepkin < ***@***.***> wrote:
pg_datanymizer.zip
<https://github.com/datanymizer/datanymizer/files/8072135/pg_datanymizer.zip>
—
Reply to this email directly, view it on GitHub
<#145 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACWRLSBGZJ57KXUMKQZ6MG3U3JSDLANCNFSM5NNBFGCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Thank you very much, I will look into it... |
|
Try the updated version, please. |
|
this one worked!
$ time ../bin/pg_datanymizer_new_debug --config ../config.yaml [...]
Fetching tables metadata...
Inspecting tables ...
Inspecting completed in 59 seconds
Inspecting table dependencies...
Inspecting completed in 0 seconds
Processing table dependencies...
Processing completed in 0 seconds
Fetching completed in 59 seconds
Sorting tables...
Sorting completed in 0 seconds
Prepare data scheme...
[1 / 373] Prepare to dump table: public.cities
…On Wed, Feb 16, 2022 at 1:20 PM Evgeniy Rashchepkin < ***@***.***> wrote:
Try the updated version, please.
pg_datanymizer.zip
<https://github.com/datanymizer/datanymizer/files/8082457/pg_datanymizer.zip>
—
Reply to this email directly, view it on GitHub
<#145 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACWRLSDQCPANUA76LTNS3GDU3PTFTANCNFSM5NNBFGCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
@martingehrke, can you check this one too please? |
|
sorry for the late reply. this one works as well;
Fetching tables metadata...
Inspecting tables ...
Inspecting completed in 1 minute
Inspecting table dependencies...
Inspecting completed in 0 seconds
Processing table dependencies...
Processing completed in 0 seconds
Fetching completed in 1 minute
Sorting tables...
Sorting completed in 0 seconds
Prepare data scheme...
…On Sun, Feb 20, 2022 at 9:53 AM Evgeniy Rashchepkin < ***@***.***> wrote:
@martingehrke <https://github.com/martingehrke>, can you check this one
too please?
pg_datanymizer.zip
<https://github.com/datanymizer/datanymizer/files/8104212/pg_datanymizer.zip>
—
Reply to this email directly, view it on GitHub
<#145 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACWRLSF5ELWY52GZ3LQY4KLU4D553ANCNFSM5NNBFGCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Fetch tables metadata...takes more time than a full database dump.
I have filters in place for only 4 tables I need out of 100+ but this still takes longer than I expect.
The text was updated successfully, but these errors were encountered: