[Core feature] Provide a way to ignore portions of the signature for cache key calculation purposes

[eapolinario]

Motivation: Why do you think this is important?

It's useful to skip some inputs for cache key calculation purposes

For example, let's say we have this task:

@task(cache=True, cache_version="v1")
def t(log_level: int, a: str) -> str:
  ...

According to the docs, one of the inputs to cache key calculation is the task signature, but in the case of this example, it'd be great if we could ignore log_level as part of the cache key calculation.

Goal: What should the final outcome look like, ideally?

We should be able to do something along the lines of:

@task(cache=True, cache_version="v1", ignore_input_vars=["log_level"])
def t(log_level: int, a: str) -> str:
    ...

This would essentially skip some of the parameters for cache key calculation purposes.

Describe alternatives you've considered

We have the ability to override the hashing mechanism used to translate python types into Flyte types, as described in https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html#caching-of-non-flyte-offloaded-objects.

One could use this idea and provide constant hashes for the arguments they want to ignore, for example:

def constant_function(x: int) -> str:
    return "const"

@task 
def t_produce_annotated_literals() -> Annotated[int, HashMethod(constant_function)]:
   log_level = ...
   return log_level

@task(cache=True, cache_version="v1")
def t(log_level: int, a: str) -> str:
   ...

@workflow
def wf() -> str:
  log_level = t_produce_annotated_literals()
  return t(log_level=log_level, a="some string")

Propose: Link/Inline OR Additional context

Expose ignore_input_vars in the @task decorator and ensure the new interface is used during cache key calculation in both local and remote executions.

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[github-actions]

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

[github-actions]

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

[troychiu]

Closing as satisfied by flyteorg/flytekit#2275 and #4618.