Skip to content

Data migration #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 29 commits into from
Jul 5, 2024
Merged

Data migration #37

merged 29 commits into from
Jul 5, 2024

Conversation

@aditya0by0
Copy link
Member

@aditya0by0 aditya0by0 commented Jul 3, 2024 •
edited
Loading

  • When initialising a dataset, the user has the option to provide a file path to csv file that contains a list of chebi ids and their assignment to a dataset (either train, validation or test). Then, instead of creating a new split, the provided split will be used
  • When initialising the dataset without providing such a file, the splits will get created automatically (as before) and the resulting split is saved as a csv file
  • When running the migration script, the chebi data files will be copied into the new structure. For the splits, the split files are combined into one file and a csv file for the split assignment will be created in addition.

aditya0by0 and others added 27 commits May 27, 2024 22:51
@aditya0by0
Data handling restructure
89ffbd3
@aditya0by0
Merge branch 'dev' into feature/testing_framework
f07f312
@aditya0by0
Update chebi tests for dynamic splits
bd6382b
@aditya0by0
Dynamic split for chebi_version_train + changes
d8abee2
@aditya0by0
Update dynamic split tests
91aa484
@aditya0by0
Update chebi + dynamic test
22f882c
@aditya0by0
Update setup.py
dde4196
@aditya0by0
Update Evaluation notebook + rel. code
aecb7e6 
Updates
- Evaluation notebook
- classification.py
- utils.py
- pre-commit + some suggestions
set split variables when required instead of during setup
98342af
remove unnecessary class instantiation
89cbdb6
Merge branch 'refs/heads/dev' into feature/testing_framework
8b22601
add isort to pre-commit, reformat with isort
b2439f8
@aditya0by0
Update .gitignore
ec6254d
@aditya0by0
remove commented out cells - eval notebook
c1b6b0d
@aditya0by0
add filename parameter to load_processed_data
667b079
@aditya0by0
Updated chebi.py for train_version restructure
8c9dfe1
@aditya0by0
minor changes in data split code
cd03023 
- removed list comprehension from data split logic
- used dataframe operations instead as they are faster
- remove looping for msss.split as no need for it, used `next` instead
@aditya0by0
Merge branch 'dev' into feature/testing_framework
0584345
fix: test for consistency across runs did validate the same run twice
f747257
@aditya0by0
migration script for chebi data for new data restructure
a87dd35
@aditya0by0
argparser + fixes
d8e68cc
@aditya0by0
transform data.pkl to data.pt instead of combining .pt split files
ae61d10
@aditya0by0
migration - raw data error fix + id col error
9c25543
@aditya0by0
pd.to_pickle instead of pickle.dump for code consistency
0c2fca1
@aditya0by0
migration : added docstring + type hints
1c4acea
@aditya0by0
logic to generate splits csv + use csv if provided
9992a15
@aditya0by0
read only first row to validate presence of relevant columns in csv
07340cb
@aditya0by0 aditya0by0 self-assigned this Jul 3, 2024
@aditya0by0 aditya0by0 requested a review from sfluegel05 July 3, 2024 18:08
@aditya0by0 aditya0by0 linked an issue Jul 3, 2024 that may be closed by this pull request
Data migration / setting fixed splits #34
Closed
@aditya0by0
Copy link
Member Author

aditya0by0 commented Jul 3, 2024

@sfluegel05 sfluegel05 marked this pull request as ready for review July 5, 2024 11:48
@sfluegel05
Copy link
Collaborator

sfluegel05 commented Jul 5, 2024

As far as I am aware, this branch originates from the brach used in PR #29. Therefore I will just merge this directly into dev.
I also added some minor changes: The cli for the migration script now uses jsonargparse, which means that one can directly use a config file instead of having to resolve the individual parameters for each class with separate arguments (this also covers other ChEBI configuration such as ChEBIOverXPartial). If specific files are not found, they are just skipped. And I added some hints for users.

@sfluegel05 sfluegel05 merged commit 7dc4e63 into dev Jul 5, 2024
@sfluegel05 sfluegel05 deleted the data_migration branch July 5, 2024 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sfluegel05 sfluegel05 sfluegel05 approved these changes

Assignees

@sfluegel05 sfluegel05

@aditya0by0 aditya0by0

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Data migration / setting fixed splits

3 participants

@aditya0by0 @sfluegel05