Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core] refactor unpacker #34

Closed
7 of 12 tasks
rudolfix opened this issue Jun 23, 2022 · 1 comment
Closed
7 of 12 tasks

[core] refactor unpacker #34

rudolfix opened this issue Jun 23, 2022 · 1 comment
Assignees
@rudolfix

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented Jun 23, 2022 •
edited
Loading

Unpacker must be refactored such as

  1. it is a class that can be instantiated several times
  2. deals with folders where files associated with load_ids reside
  3. uses parser configured in the schema
  4. uses naming convention configured in the schema (normalization, path generation for tables and nested entities etc.)
  5. implements PoolRunnable to be able to be run
  6. many instances of Prometheus metrics in different namespaces may exist
  7. processes data from jsonl one document at a time
  8. for tables that are marked as merge or replace it must merge the files corresponding to single table

this also means
@rudolfix rudolfix self-assigned this Jun 23, 2022
@rudolfix rudolfix mentioned this issue Jun 23, 2022
Closed
3 tasks
@rudolfix rudolfix changed the title [core] make parser and name normalization function pluggable in unpacker [core] refactor unpacker Jun 24, 2022
@rudolfix rudolfix mentioned this issue Aug 9, 2022
Merged
@rudolfix
Copy link
Collaborator Author

closing due to 90% of tasks done. the issue with replace requires separate ticket

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rudolfix rudolfix

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rudolfix