Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running replibyte results in command being killed #244

Open
pKorsholm opened this issue Dec 9, 2022 · 0 comments
Open

Running replibyte results in command being killed #244

pKorsholm opened this issue Dec 9, 2022 · 0 comments

Comments

@pKorsholm
Copy link

pKorsholm commented Dec 9, 2022

I'm currently running replibyte trying to subset my database.

My config is something like:

source:
  connection_uri: $DB_URL
  database_subset:
    database: public
    table: cart
    strategy_name: random
    strategy_options:
      percent: 20
    passthrough_tables: 
      - country
      .... 
      <30 more tables> 
  transformers: 
    - database: public
      table: customer
      columns:
        - name: email
          transformer_name: email
    ... 
    <23 columns and 8 tables>    

This results in replibyte running out of memory (I'm guessing given the screenshot of the process from my activity montitor below, taken right before the process crashed)

image

Is there any way to reduce the memory consumption, the database is in total around 250mb (148mb if asking postgres with: SELECT pg_size_pretty( pg_database_size('dbname') )) and 51gb of memory taken up seems excessive.

@pKorsholm pKorsholm reopened this Dec 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant