S-Prompts for ViT and Text Transformers #388
Conversation
| @@ -41,7 +41,10 @@ using Renate (e.g., using :py:func:`~renate.training.training.run_training_job`; | |||
| - A class that implements a Learning to Prompt method for ViTs. The methods trains only the input prompts that are sampled from a prompt pool in an input dependent fashion. | |||
| * - ``"LearningToPromptReplay"`` | |||
| - :py:class:`LearningToPromptLearner <renate.updaters.experimental.l2p.LearningToPromptReplayLearner>` | |||
| - A class that extends the Learning to Prompt method to use a memory replay method like "Offline-ER" | |||
| - A class that extends the Learning to Prompt method to use a memory replay method like "Offline-ER". | |||
There was a problem hiding this comment.
since this is about supported algorithms, it should list S-Prompts and give guidance how to use it with SPeft.
| @@ -367,6 +367,8 @@ def __init__( | |||
| def prepare_data(self) -> None: | |||
| """Download DomainNet dataset for given domain.""" | |||
| file_name = f"{self.data_id}.zip" | |||
| # update dataset name: | |||
| self._dataset_name = self.data_id | |||
There was a problem hiding this comment.
why is this here? prepare_data is called only once. therefore, one could set this already as part of the constructor. should this replace L365?
| self._M = prompt_size | ||
| self._task_id = task_id | ||
| self._per_task_classifier = per_task_classifier | ||
| logger.warning(f"Task id is {self._task_id}") |
| self._backbone["transformer"].requires_grad_(False) | ||
| self._backbone["prompt_pool"].requires_grad_(True) | ||
|
|
||
| # self._backbone["transformer"].transformer._backbone.enable_gradient_checkpointing() |
| # self.s_prompts | ||
| self._backbone["prompt_pool"].increment_task() | ||
|
|
||
| def forward_for_monkey_patching( |
| ] | ||
| ) | ||
| else: | ||
| logits = self._backbone["classifier"]["0"](features) |
There was a problem hiding this comment.
can we remove the hard-coding of "0" somehow? what is defining that name?
There was a problem hiding this comment.
It is the update/task_id variable converted into a string. 0 just implies that we are in the first
update. Removing it will just be a cosmetic first_task_name = "0", which doesn't seem to serve any
purpose.
| Args: | ||
| in_features: size of each input sample | ||
| out_features: size of each output sample | ||
| bias: If set to ``False``, the layer will not learn an additive bias. |
There was a problem hiding this comment.
missing documentation of args
| } | ||
| super().__init__(all_layers) | ||
|
|
||
| def increment_task(self) -> None: |
There was a problem hiding this comment.
is there a nice way to reuse this function in the constructor to populate all_layers?
| self, epoch: int, batch_idx: int, optimizer: Optimizer, optimizer_idx: int | ||
| ) -> None: | ||
| """Explicitly setting grads to None instead of zero.""" | ||
| optimizer.zero_grad(set_to_none=True) |
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.