Persistence #1017

Eldinnie · 2018-02-22T10:18:54Z

Adding a construct for adding persistence to PTB.

Added Basepersistence as an interface for persistence classes
Added PicklePersistence for persistence using pickle
Added DictPersistence for persistence using dict/JSON
Integrating the persistence into the dispatcher, conversationhandler and updater classes
Added helper methods for (de)serializing conversations and user/chat_data

Add tests for Basepersistence

For Py2 compatibility

jsmnbom · 2018-03-22T20:52:09Z

@Eldinnie Is this ready for review? Did we discuss it in the chat and I just forgot about it?

JosXa · 2018-04-05T00:48:51Z

Hi @SehgalDivij, this is an implementation that will generally solve the persistence for ptb. An abstract base class was provided that will allow users to implement their own serialization interfaces and pass them on to the library, effectively giving you the ability to serialize almost every collection in the library.

jsmnbom

Adding some comments, mostly relating to the overall implementation. I have not yet read the code through thoroughly :)

jsmnbom · 2018-04-30T17:21:12Z

telegram/ext/basepersistence.py

+    Attributes:
+        store_user_data (:obj:`bool`): Optional, Whether user_data should be saved by this
+            persistence class.
+        store_chat_data (:obj:`bool`): Optional. Whether user_data should be saved by this


jsmnbom · 2018-04-30T17:21:25Z

telegram/ext/basepersistence.py

+    Args:
+        store_user_data (:obj:`bool`, optional): Whether user_data should be saved by this
+            persistence class. Default is ``True``.
+        store_chat_data (:obj:`bool`, optional): Whether user_data should be saved by this


jsmnbom · 2018-04-30T17:22:44Z

telegram/ext/basepersistence.py

+        persistence a chance to finish up saving or close a database connection gracefully. If this
+        is not of any importance just pass will be sufficient.
+        """
+        raise NotImplementedError


If this is not of importance, should it really raise NYE?

jsmnbom · 2018-04-30T17:23:25Z

telegram/ext/conversationhandler.py

@@ -76,6 +76,10 @@ class ConversationHandler(Handler):
        per_user (:obj:`bool`): Optional. If the conversationkey should contain the User's ID.
        per_message (:obj:`bool`): Optional. If the conversationkey should contain the Message's
            ID.
+        name (:obj:`str`): Optional. The name for this conversationhandler. Required for


Could we default to an internal counter or something?

That would rely on the conv_handlers keeping in the same order when the code is changing. Better to force naming IMO

jsmnbom · 2018-04-30T17:24:15Z

telegram/ext/conversationhandler.py

@@ -76,6 +76,10 @@ class ConversationHandler(Handler):
        per_user (:obj:`bool`): Optional. If the conversationkey should contain the User's ID.
        per_message (:obj:`bool`): Optional. If the conversationkey should contain the Message's
            ID.
+        name (:obj:`str`): Optional. The name for this conversationhandler. Required for
+            persistence
+        persistent (:obj:`bool`): Optional. If the conversations dict for this handler should be


If name is required, could we not just have a single persistent_name that turns persistence on if a string?

Interesting. But this way we also have the name seperate if not persistent. Might be usefull some day?

jsmnbom · 2018-04-30T17:26:42Z

telegram/ext/dispatcher.py

+            if self.persistence.store_chat_data:
+                self.chat_data = self.persistence.get_chat_data()
+                if not isinstance(self.chat_data, defaultdict):
+                    raise ValueError("`chat_data must be of type defaultdict")


misplaced `

telegram/ext/dispatcher.py

jsmnbom · 2018-04-30T17:28:58Z

telegram/ext/picklepersistence.py

+from telegram.ext import BasePersistence
+
+
+class PicklePersistence(BasePersistence):


Is there ever a file where it makes sense to use single_file=False?

dunno, but it might

I'm not really a fan of having features that we aren't sure anyone can use... It's just more maintenance work for us :/

Also you can't choose the filetype (suffix) when using single_file=False.?

No, but I don;t really see a problem here. I think it's versatile enough this way.

telegram/ext/picklepersistence.py

jsmnbom

Few more comments, but otherwise I think this is probably good to go. Do we wanna merge it after or before #1100 ?

jsmnbom · 2018-05-15T13:22:15Z

telegram/ext/updater.py

@@ -485,6 +492,8 @@ def _join_threads(self):

    def signal_handler(self, signum, frame):
        self.is_idle = False
+        if self.persistence:


This should probably be moved into the if self.running: block

Cause otherwise we impede the ability to exit immediately? Like if the user presses CTRL-C (or whatever) twice, it shouldn't save persistence twice, it just quit ASAP imo

jsmnbom · 2018-05-15T13:23:06Z

telegram/ext/dispatcher.py

+        else:
+            self.persistence = None
+
+        if self.persistence:


Is there a reason this block isn't just in the if persistence: block above?

there was no good one

Eldinnie · 2018-09-18T10:02:49Z

Ok for future reference (And for the future CHANGES.RST)
Beside the obvious persistence classes and construct in this PR I added 3 helper methods to telegram.utils.helpers namely:

enocde_conversations_to_json(conversations)
to json enocde a conversations dict which uses tuples as keys (which give a ValueError when trying to encode as JSON. This helper JSON-serializes the tuples to before serializing the dict.
decode_conversations_from_json(json_string)
To reverse the operation from the encoding. It takes a string as encoded by the previous function and returns a conversations dict as the conversationhandler expects it
decode_user_chat_data_from_json(data)
When JSON-serializing user/chat_data to json the keys (which are ints) are converted to strings.
This function tries to cast dicts keys to ints.

jsmnbom

Overall this looks brilliant!
I added a few comments with some suggestions and questions in places where I got a bit confused.

The only quib I really have implementation-wise, is that Persistence.update_* all receive the entire defaultdict, instead of only the changed part, but I'm not sure if there is really a good way to change this. The reason I'd want this btw, is that if you were to create a DB backend like mysql or redis or whatever, then that's probably what you'd want.

jsmnbom · 2018-09-18T11:04:21Z

telegram/ext/basepersistence.py

+      :meth:`update_user_data`.
+    * If you want to store conversation data with :class:`telegram.ext.ConversationHandler`, you
+      must overwrite :meth:`get_conversations` and :meth:`update_conversations`.
+    * :meth:`flush` will be called when the bot is shutdown, and must always be overwritten.


This docstring should probably be changed, since flush mustn't always be overwritten as it is implemented atm (which I think is a good thing)

telegram/ext/dictpersistence.py

jsmnbom · 2018-09-18T11:08:54Z

telegram/ext/dispatcher.py

-        """:obj:`dict`: A dictionary handlers can use to store data for the chat."""
+        if persistence:
+            if not isinstance(persistence, BasePersistence):
+                self.logger.warning("persistence should be based on telegram.ext.BasePersistence")


Imo this should raise an exception

I thought about that. But if someone really wants to create their own version without basing it on Basepersistence they should be allowed

I equate it to how we handle handlers in dispatche https://github.com/python-telegram-bot/python-telegram-bot/blob/master/telegram/ext/dispatcher.py#L329
Also BaseFilter iirc

ok seems legit

jsmnbom · 2018-09-18T11:11:21Z

telegram/ext/picklepersistence.py

+from telegram.ext import BasePersistence
+
+
+class PicklePersistence(BasePersistence):


I'm not really a fan of having features that we aren't sure anyone can use... It's just more maintenance work for us :/

telegram/ext/picklepersistence.py

jsmnbom · 2018-09-18T11:16:22Z

telegram/ext/picklepersistence.py

+from telegram.ext import BasePersistence
+
+
+class PicklePersistence(BasePersistence):


Also you can't choose the filetype (suffix) when using single_file=False.?

telegram/utils/helpers.py

jsmnbom · 2018-09-18T11:23:31Z

tests/test_persistence.py

+        u = Updater(bot=bot, persistence=pickle_persistence_2)
+        dp = u.dispatcher
+        dp.add_handler(h2)
+        dp.process_update(update)


Should this check to make sure that first and second even got called?

@jsmnbom

As discussed with @jsmnbom

Eldinnie · 2018-09-18T21:15:54Z

only pypy failing on Travis.
Coverage can be checked here. Diff coverage is 94.04%
Please check if you're happy now @jsmnbom and others welcome for review (@tsnoam @jh0ker )

jsmnbom

We should probably add some help to the wiki shortly after merge.
Maybe even an example?

Overall really amazing work @Eldinnie ! :D

owneroxxor · 2018-10-04T04:42:42Z

Already using PTB 11.1.0 but my script gives error when trying to import BasePersistence fromtelegram.ext.
Is persistence feature already launched?

Eldinnie · 2018-10-04T06:09:24Z

@owneroxxor it's merged into master, but not yet released. In the next released version it will be available.

alisherafat01 · 2018-10-26T14:49:13Z

@Eldinnie when do you release this feature? we need this!

davidh- · 2018-10-27T03:02:41Z

@Eldinnie when do you release this feature? we need this!

Also would like a release! I guess for the meantime we can install from master.

Eldinnie added 9 commits February 20, 2018 11:34

BasePersistence

492422d

basic construct

b51207c

Keep working

03056e1

Continue work

f7e81b9

Add tests for Basepersistence

Finish up BasePersistence and implementation

ecf88fe

PickelPersistence and start tests

4f19acf

Finishing up

8b3b93c

Oops, left in some typings

c5f703a

Compatibilty issues regarding py2 solved

0fd9a4e

For Py2 compatibility

Eldinnie changed the title ~~[DO NOT MERGER] Persistence Construct~~ [DO NOT MERGE] Persistence Construct Feb 22, 2018

increasing coverage

65b8d65

Eldinnie changed the title ~~[DO NOT MERGE] Persistence Construct~~ Persistence Mar 1, 2018

JosXa mentioned this pull request Apr 4, 2018

Feature Request: Persist data between responses in ConversationHandler #385

Closed

jsmnbom reviewed Apr 30, 2018

View reviewed changes

jsmnbom mentioned this pull request Apr 30, 2018

Add Menu proof of concept #639

Closed

Eldinnie added 2 commits May 3, 2018 15:50

Small changes due to CR

dce3109

Merge branch 'master' into presistence

2954379

jsmnbom reviewed May 15, 2018

View reviewed changes

Eldinnie and others added 5 commits September 17, 2018 09:00

Merge branch 'master' into presistence

4ff74e6

All persistence tests in one file

c1f23b1

add DictPersistence

bd75304

Last changes per CR

1201aa6

forgot change

b3ceafe

jsmnbom requested changes Sep 18, 2018

View reviewed changes

Eldinnie added 2 commits September 18, 2018 13:54

changes per CR

b1544a9

call update_* only with relevant data

7e0feda

As discussed with @jsmnbom

jsmnbom approved these changes Sep 18, 2018

View reviewed changes

Eldinnie added 2 commits September 19, 2018 12:40

Add conversationbot Example

27c458d

should not have committed API-key

6ff48c0

jsmnbom merged commit 4397903 into master Sep 20, 2018

Eldinnie deleted the presistence branch September 21, 2018 12:29

Eldinnie mentioned this pull request Oct 2, 2018

PTB on Serverless Architectures #1118

Closed

jsmnbom mentioned this pull request Oct 29, 2018

Add Persistence #638

Closed

github-actions bot locked and limited conversation to collaborators Aug 21, 2020

		from telegram.ext import BasePersistence


		class PicklePersistence(BasePersistence):

Persistence #1017

Persistence #1017

Conversation

Eldinnie commented Feb 22, 2018 • edited

jsmnbom commented Mar 22, 2018 • edited

JosXa commented Apr 5, 2018

jsmnbom left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsmnbom left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eldinnie commented Sep 18, 2018

jsmnbom left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eldinnie commented Sep 18, 2018 • edited

jsmnbom left a comment

Choose a reason for hiding this comment

owneroxxor commented Oct 4, 2018

Eldinnie commented Oct 4, 2018

alisherafat01 commented Oct 26, 2018

davidh- commented Oct 27, 2018

Eldinnie commented Feb 22, 2018 •

edited

jsmnbom commented Mar 22, 2018 •

edited

Eldinnie commented Sep 18, 2018 •

edited