Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Support self-supervised pre-training #3

Closed
athewsey opened this issue Nov 3, 2021 · 0 comments
Closed

[Enhancement] Support self-supervised pre-training #3

athewsey opened this issue Nov 3, 2021 · 0 comments
Assignees
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@athewsey
Copy link
Contributor

athewsey commented Nov 3, 2021

Notebook 2 currently has a placeholder section discussing how model accuracy might be improved by self-supervised pre-training (such as masked language modelling) on a broader corpus of unlabelled data, before fine-tuning on the end task with annotations.

However, the model training scripts in notebooks/src aren't set up for that yet.

It'd be great if we can flesh this out to a point where users can optionally run a pre-training job with Textract JSON only (ideally still with the ability to start from a pre-trained model from HF model zoo); and then use that training job instead of the model zoo models, as the starting point for the fine-tuning task.

@athewsey athewsey added enhancement New feature or request good first issue Good for newcomers labels Nov 3, 2021
athewsey added a commit that referenced this issue Nov 17, 2021
Refactor LayoutLM model training & inference code to support MLM
pre-training before NER fine-tuning, and present steps in notebook.
Upgrade HF container versions 4.6->4.11.

Draft for #3
@athewsey athewsey self-assigned this Nov 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant