New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MCC value problem on Nucleotide Transformer #31
Comments
|
Not sure what else to tell you, if you follow the previous comment it will
calculate the mcc.
If there’s some other custom functionality you need then you’ll need to get
very familiar with Pytorch Lightning. Good luck!
…On Fri, Nov 24, 2023 at 2:31 PM WeiminWu2000 ***@***.***> wrote:
Thanks for your reply. I am running the default repo. Yes, I learn what
you mean, but I want to calculate the MCC value and output it. I think
maybe it does not calculate or log the MCC, could I modify something to
output the MCC other than the accuracy (it works well when I use accuracy)
Besides, I want to perform a train-val-test experiment on dataset
Nucleotide Transformer. In the default setting, the code onlys performs
train and test without validating. I use the code on line 675 in train.py,
(i.e. trainer.test(model)), but it has the following problem:
Traceback (most recent call last):
File "/root/code/dnabert3/hyena-dna/train.py", line 690, in main
train(config)
File "/root/code/dnabert3/hyena-dna/train.py", line 675, in train
trainer.test(model)
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py",
line 780, in test
return call._call_and_handle_interrupt(
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py",
line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py",
line 829, in _test_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py",
line 1098, in _run
results = self._run_stage()
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py",
line 1174, in _run_stage
return self._run_evaluate()
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py",
line 1214, in _run_evaluate
eval_loop_results = self._evaluation_loop.run()
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py",
line 199, in run
self.advance(*args, **kwargs)
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py",
line 152, in advance
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches,
kwargs)
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py",
line 199, in run
self.advance(*args, **kwargs)
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py",
line 137, in advance
output = self._evaluation_step(**kwargs)
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py",
line 234, in _evaluation_step
output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py",
line 1480, in _call_strategy_hook
output = fn(*args, **kwargs)
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py",
line 399, in test_step
return self.model.test_step(*args, **kwargs)
File "/root/code/dnabert3/hyena-dna/train.py", line 440, in test_step
return self._shared_step(
File "/root/code/dnabert3/hyena-dna/train.py", line 338, in _shared_step
torchmetrics = getattr(self, f'{prefix}_torchmetrics')
File
"/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/torch/nn/modules/module.py",
line 1265, in *getattr*
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'SequenceLightningModule' object has no attribute
'final/val_torchmetrics'
Could you please help me to solve them, thank you very much!
—
Reply to this email directly, view it on GitHub
<#31 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGNEKPCDNXEIUGEVI6KLCTTYGEN4ZAVCNFSM6AAAAAA7ZMXDP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGEYTMOBXGE>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Thanks very much. I'll learn about it. |
Actually, what I showed above grabs the default metric set by the Nucleotide Transformer benchmarks, so it could be acc, or mcc. To force it show mcc, you can hardcode it, like this. Note, this is what Lightning uses to save the best checkpoints, it "monitors" the val/mcc and will save the "max" or highest mcc ckpts.
|
Thanks for your reply! I'll try it! |
looks like that worked |
Yes, thanks very much for you help! |
I really appreciate your work, but when I train model on the downstream task Nucleotide Transformer, it shows that no MCC result returned, the details are as follows, could you please help me to solve it, thank you very mcuh!
Traceback (most recent call last):
File "/root/code/dnabert3/hyena-dna/train.py", line 692, in main
train(config)
File "/root/code/dnabert3/hyena-dna/train.py", line 673, in train
trainer.fit(model)
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
call._call_and_handle_interrupt(
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1098, in _run
results = self._run_stage()
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1177, in _run_stage
self._run_train()
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1200, in _run_train
self.fit_loop.run()
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
self.on_advance_end()
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 295, in on_advance_end
self.trainer._call_callback_hooks("on_train_epoch_end")
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1380, in _call_callback_hooks
fn(self, self.lightning_module, *args, **kwargs)
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 312, in on_train_epoch_end
self._save_topk_checkpoint(trainer, monitor_candidates)
File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 367, in _save_topk_checkpoint
raise MisconfigurationException(m)
lightning_lite.utilities.exceptions.MisconfigurationException:
ModelCheckpoint(monitor='val/mcc')
could not find the monitored key in the returned metrics: ['trainer/loss', 'trainer/epoch', 'val/accuracy', 'val/loss', 'train/accuracy', 'train/loss', 'epoch', 'step']. HINT: Did you calllog('val/mcc', value)
in theLightningModule
?The text was updated successfully, but these errors were encountered: