Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat]Add Redis backend feature to TFRA #157

Merged
merged 1 commit into from
Sep 14, 2021
Merged

Conversation

MoFHeka
Copy link
Contributor

@MoFHeka MoFHeka commented Sep 14, 2021

Add Redis backend feature to TFRA

Description

Brief Description of the PR:

Added Redis as the backend for KV storage and changed the Python interface by calling the Config class and creating/passing the KvCreator class to fit the different backend. Also setting backend parameters with json file.

Remember the devices of TFRA-Redis should be set as local, not PS!

Type of change

  • Bug fix
  • New Tutorial
  • Updated or additional documentation
  • Additional Testing
  • New Feature

Checklist:

  • I've properly formatted my code according to the guidelines
    • By running yapf
    • By running clang-format
  • This PR addresses an already submitted issue for TensorFlow Recommenders-Addons
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works

How Has This Been Tested?

redis_config1=tfra.dynamic_embedding.RedisTableConfig(
    redis_config_abs_dir=/tmp/test/redis_config.json”
)
redis_creator1=tfra.dynamic_embedding.RedisTableCreator(redis_config1)

self.user_embeddings = tfra.dynamic_embedding.get_variable(
            name="user_dynamic_embeddings",
            dim=self.embedding_size,
            initializer=tf.keras.initializers.RandomNormal(-1, 1),
            checkpoint=False,
            KVCreator=redis_creator1
            )

or you can do it like this:

redis_config_dir = os.path.join(tempfile.mkdtemp(dir=os.environ.get('TEST_TMPDIR')), "save_restore")
redis_config_path = os.path.join(tempfile.mkdtemp(prefix=redis_config_dir), "hash")
os.makedirs(redis_config_path)
redis_config_path = os.path.join(redis_config_path, "redis_config.json")
redis_config_params = {
  "redis_host_ip":["127.0.0.1"],
  "redis_host_port":[6379],
  "using_model_lib":False
}
with open(redis_config_path, 'w', encoding='utf-8') as f:
  f.write(json.dumps(redis_config_params, indent=2, ensure_ascii=True))
redis_config = de.RedisTableConfig(
  redis_config_abs_dir=redis_config_path
)
redis_creator=tfra.dynamic_embedding.RedisTableCreator(redis_config)

self.user_embeddings = tfra.dynamic_embedding.get_variable(
            name="user_dynamic_embeddings",
            dim=self.embedding_size,
            initializer=tf.keras.initializers.RandomNormal(-1, 1),
            checkpoint=False,
            KVCreator= redis_creator
            )

How Has This Been Used?

Please visit this folder:
demo/dynamic_embedding/movielens-100k-estimator-redis

try to avoid github action restriction without any CI, pls do it by yourself.
@rhdong
Copy link
Member

rhdong commented Sep 14, 2021

LGTM

@rhdong rhdong merged commit cda023d into tensorflow:master Sep 14, 2021
@rhdong
Copy link
Member

rhdong commented Sep 14, 2021

Great, thank you HeJia! @MoFHeka

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants