Fix batch processing #26

nurikk · 2021-07-01T07:15:21Z

This pr fixes issue with processing huge tables. It calls import_data inside loop that calls fetchmany and clears data and StringIO buffers after each iteration

Depends on #25

… growth

hkage · 2021-07-02T08:04:54Z

Hi,

I tested your changes against a ~1,6 GB sized database and tables with at most ~370k data entries (per table). The anonymization breaks at some point with an OutOfMemory error. Have you discovered similar issues?

Im an running the anonymization with Python 3.9 in a virtualenv and on a Ubuntu Linux 21.04. with 24 GB of RAM.

postgres=# SELECT version();
                                                           version                                                            
------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 10.7 (Ubuntu 10.7-0ubuntu0.18.10.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 8.2.0-7ubuntu1) 8.2.0, 64-bit
(1 row)

Anonymizing |█                               | 10000/292059Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/__main__.py", line 18, in <module>
    main()
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/__main__.py", line 10, in main
    main()
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/cli.py", line 71, in main
    anonymize_tables(connection, schema.get('tables', []), verbose=args.verbose)
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/utils.py", line 40, in anonymize_tables
    build_and_then_import_data(connection, table_name, primary_key, columns, excludes,
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/utils.py", line 83, in build_and_then_import_data
    import_data(connection, columns, table, primary_key, data)
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/utils.py", line 158, in import_data
    cursor.execute('DROP TABLE %s;' % temp_table)
psycopg2.errors.OutOfMemory: out of shared memory
HINT:  You might need to increase max_locks_per_transaction.

nurikk · 2021-07-02T08:26:23Z

Hi,

I tested your changes against a ~1,6 GB sized database and tables with at most ~370k data entries (per table). The anonymization breaks at some point with an OutOfMemory error. Have you discovered similar issues?

Im an running the anonymization with Python 3.9 in a virtualenv and on a Ubuntu Linux 21.04. with 24 GB of RAM.

postgres=# SELECT version();
                                                           version                                                            
------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 10.7 (Ubuntu 10.7-0ubuntu0.18.10.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 8.2.0-7ubuntu1) 8.2.0, 64-bit
(1 row)

Anonymizing |█                               | 10000/292059Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/__main__.py", line 18, in <module>
    main()
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/__main__.py", line 10, in main
    main()
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/cli.py", line 71, in main
    anonymize_tables(connection, schema.get('tables', []), verbose=args.verbose)
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/utils.py", line 40, in anonymize_tables
    build_and_then_import_data(connection, table_name, primary_key, columns, excludes,
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/utils.py", line 83, in build_and_then_import_data
    import_data(connection, columns, table, primary_key, data)
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/utils.py", line 158, in import_data
    cursor.execute('DROP TABLE %s;' % temp_table)
psycopg2.errors.OutOfMemory: out of shared memory
HINT:  You might need to increase max_locks_per_transaction.

No, i'm not getting such issues. Without proposed changes I assume everything works fine?

hkage · 2021-07-02T08:37:09Z

Hi,
I tested your changes against a ~1,6 GB sized database and tables with at most ~370k data entries (per table). The anonymization breaks at some point with an OutOfMemory error. Have you discovered similar issues?
Im an running the anonymization with Python 3.9 in a virtualenv and on a Ubuntu Linux 21.04. with 24 GB of RAM.

postgres=# SELECT version();
                                                           version                                                            
------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 10.7 (Ubuntu 10.7-0ubuntu0.18.10.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 8.2.0-7ubuntu1) 8.2.0, 64-bit
(1 row)

Anonymizing |█                               | 10000/292059Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/__main__.py", line 18, in <module>
    main()
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/__main__.py", line 10, in main
    main()
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/cli.py", line 71, in main
    anonymize_tables(connection, schema.get('tables', []), verbose=args.verbose)
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/utils.py", line 40, in anonymize_tables
    build_and_then_import_data(connection, table_name, primary_key, columns, excludes,
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/utils.py", line 83, in build_and_then_import_data
    import_data(connection, columns, table, primary_key, data)
  File "/home/henning/Projekte/postgresql-anonymizer/pganonymizer/utils.py", line 158, in import_data
    cursor.execute('DROP TABLE %s;' % temp_table)
psycopg2.errors.OutOfMemory: out of shared memory
HINT:  You might need to increase max_locks_per_transaction.

No, i'm not getting such issues. Without proposed changes I assume everything works fine?

Yes, as soon as I change to the development branch the anonymization works fine and completes after a couple of minutes.

I haven't take a deeper look into the max_locks_per_transaction settings from PostgreSQL, but it seems like the number of locks during the anonymization exceeds PostgreSQL's default. I tried to split the cursor operations into multiple transactions, but that resulted in errors when deleting the temporary tables (at https://github.com/rheinwerk-verlag/postgresql-anonymizer/blob/development/pganonymizer/utils.py#L157).

nurikk · 2021-07-02T08:46:32Z

I'll fix this issue

hkage · 2021-07-02T09:03:25Z

I'll fix this issue

Thanks. I can support you with additional debug / system information if that helps you.

nurikk · 2021-07-02T10:06:30Z

Hm, I can't reproduce it on my test db
Can you try to test against this pr? It's already includes changes from this PR and might resolves psycopg2.errors.OutOfMemory: out of shared memory issue

nurikk · 2021-07-05T04:33:40Z

Superseded by #27

hkage · 2021-07-05T06:34:30Z

Hm, I can't reproduce it on my test db
Can you try to test against this pr? It's already includes changes from this PR and might resolves psycopg2.errors.OutOfMemory: out of shared memory issue

Sure, I will check it with PR 27. Thanks!

Ainur Timerbaev added 2 commits July 1, 2021 13:06

Remove column specification for cursor.copy_from call

3f17efc

Call import_data within cursor.fetchmany loop, to prevent used memory…

15f5ce9

… growth

nurikk mentioned this pull request Jul 2, 2021

Better anonymisation #27

Merged

nurikk closed this Jul 5, 2021

becurrie mentioned this pull request Nov 24, 2021

parmap no longer supports Python 2.7 #35

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix batch processing #26

Fix batch processing #26

nurikk commented Jul 1, 2021

hkage commented Jul 2, 2021 •

edited

nurikk commented Jul 2, 2021 •

edited

hkage commented Jul 2, 2021 •

edited

nurikk commented Jul 2, 2021

hkage commented Jul 2, 2021

nurikk commented Jul 2, 2021 •

edited

nurikk commented Jul 5, 2021

hkage commented Jul 5, 2021

Fix batch processing #26

Fix batch processing #26

Conversation

nurikk commented Jul 1, 2021

hkage commented Jul 2, 2021 • edited

nurikk commented Jul 2, 2021 • edited

hkage commented Jul 2, 2021 • edited

nurikk commented Jul 2, 2021

hkage commented Jul 2, 2021

nurikk commented Jul 2, 2021 • edited

nurikk commented Jul 5, 2021

hkage commented Jul 5, 2021

hkage commented Jul 2, 2021 •

edited

nurikk commented Jul 2, 2021 •

edited

hkage commented Jul 2, 2021 •

edited

nurikk commented Jul 2, 2021 •

edited