Fixes: #2426 Populate with UserPresences #2626

Juanvulcano · 2016-12-07T01:19:17Z

Hi, debugging models is harder than I thought it would haha.
Note that currently there is an error that fails the Mypy test, I would love to know what annotations are needed to solve it.

smarx · 2016-12-07T01:19:18Z

Automated message from Dropbox CLA bot

@Juanvulcano, it looks like you've already signed the Dropbox CLA. Thanks!

umairwaheed · 2016-12-07T06:46:23Z

zilencer/management/commands/populate_db.py

@@ -177,8 +179,22 @@ def handle(self, **options):
                                 Recipient.objects.filter(type=Recipient.STREAM)]

        # Extract a list of all users
-        user_profiles = [user_profile.id for user_profile in UserProfile.objects.all()] # type: List[int]
-
+        user_profiles = [user_profile for user_profile in UserProfile.objects.all()] # type: List[Any]


You can use user_profiles = list(UserProfile.objects.all()). The type should be UserProfile instead of Any.

umairwaheed · 2016-12-07T06:48:08Z

zilencer/management/commands/populate_db.py

+
+        # Populate users with some bar data
+        for user in user_profiles:
+            month = random.choice(range(1, 13)) # type: Any


Use random.randint(1, 12). Type should be int. Same for other places where you have used choice.

umairwaheed · 2016-12-07T06:49:34Z

zilencer/management/commands/populate_db.py

+                random_client = get_client("API")
+            else:
+                random_client = get_client("website")
+            UserPresence.objects.get_or_create(user_profile=user, client=random_client, timestamp=random_date, status=random_status)


Use just status, date and client, drop the random_` part.

umairwaheed · 2016-12-07T06:51:58Z

zilencer/management/commands/populate_db.py

+                random_client = get_client("website")
+            UserPresence.objects.get_or_create(user_profile=user, client=random_client, timestamp=random_date, status=random_status)
+
+        user_profiles = [user_profile.id for user_profile in UserProfile.objects.all()]


No need to hit the DB again, just use user_profile_ids = [user_profile.id for user_profile in user_profiles].

umairwaheed · 2016-12-07T06:54:51Z

@Juanvulcano, added some feedback, taking care of it should fix your mypy issues as well. I would recommend that you take your mypy changes that are not related to this issue to a separate commit. Also, add Fixes: #2426 in your commit message. Thanks for working on this :).

Juanvulcano · 2016-12-07T21:20:58Z

Hi, I followed the recommended suggestions but Mypy is still failling, what could be wrong?

Juanvulcano · 2016-12-07T23:31:24Z

Umar suggests changing UserProfile model to be of UserProfile type instead of List[Any], this gives the following error:
https://travis-ci.org/zulip/zulip/jobs/182114719

Also, test is failing here I think because the data I am delivering is not the same but I don't know how to fix it. Maybe I need to add new list to the test_runner.py test? https://travis-ci.org/zulip/zulip/jobs/182114718

mjec · 2016-12-08T01:30:12Z

Hi @Juanvulcano, this is looking really good! I think to fix up this remaining Travis CI failure you'll need to edit zerver/tests/test_presence.py line 68 to set the expected data for self.assertEqual(list(json['presences'].keys()), ...

It'd also be great if you could reword your commits so that there are only two of them, even after the tests fix described above. Then:

your first commit should include the fix to test_presence.py and the message should include "Fixes: populate_db should generate some default presence data #2426"
your second commit can just say "Fix mypy annotations" and doesn't need to refer to 2426

Once this is passing tests I expect it will be good to merge!

Juanvulcano · 2016-12-08T21:49:28Z

Hi, I've been trying to debug this assertion error.
It looks like everytime I run this it gets the data in a random order, but data is not generated randomly.
How can I fix this? Maybe UserProfile.objects.all() gets the users with a different order everytime?
or it might be needed to filter self.assertEqual(list(json['presences'].keys()) ?

https://travis-ci.org/zulip/zulip/jobs/182406300

https://travis-ci.org/zulip/zulip/jobs/182413140

timabbott · 2016-12-08T21:51:12Z

What's going on here is that we some tests specifically for the presence system which assume that they created the only UserPresence entries that exist, and they fail when you add in some extra random data.

We really only need your pre-populated presence data for manual testing, not for the automated test suite.

@Juanvulcano can you try putting your new presence creation stuff in populate_db inside an if not options["test_suite"]: check?

I think the other thing we could consider is hardcoding the data rather than generating it randomly.

timabbott · 2016-12-08T21:54:53Z

(accidentally hit enter on the previous message; I just edited my comment to not be half-finished)

Juanvulcano · 2016-12-08T22:12:31Z

Hi @timabbott , just before reading your message I commited some changes that made the automated test suite work with the new data. Do you think it's a good solution or would you prefer me to have the new data inside the if statement?

timabbott · 2016-12-08T22:21:43Z

Well, my guess is the new data will probably change every time you run tools/do-destroy-rebuild-test-database (run as part of provision if the schema has changed), but depending on the changes that could be reasonable.

timabbott · 2016-12-08T22:22:18Z

Looking at the test changes, I think I'd rather add the new data inside the if statement.

Juanvulcano · 2016-12-08T22:33:34Z

@timabbott Data was not randomly generated and was sorted before assessing, why do you think the new data would change?

umairwaheed · 2016-12-09T07:29:55Z

zilencer/management/commands/populate_db.py

+                date = datetime.datetime.today() # type: datetime.datetime
+                client = get_client("website")
+                for i in range(3):
+                    client = get_client("API")


You are overriding client here. client will never be equal toget_client("website") due to client = get_client("API")

I guess you can just remove the following block:

for i in range(3): client = get_client("API")

umairwaheed · 2016-12-09T07:41:23Z

@Juanvulcano, looking great. So it would be great if you take all the mypy fixes that are not related to #2426 in another commit. You also need to update the commit message. It should have the following format:

[Brief description]  # see the title of the issue

Fixes #2426

You should have two commits at the end. One commit with unrelated mypy changes and the other with the fix.

showell · 2016-12-09T19:42:33Z

For your first commit can you say something like this?:

Add UserPresence rows to test databases.

Fixes #2426

showell · 2016-12-09T20:04:26Z

zilencer/management/commands/populate_db.py

+            # Populate users with some bar data
+            for user in user_profiles:
+                status = 1 # type: int
+                date = datetime.datetime.today() # type: datetime.datetime


use now() here instead of today()

Fixes: zulip#2426 mypy: Added annotations to populate_db.py

showell · 2016-12-09T23:41:36Z

Merged! Thanks @Juanvulcano!

timabbott · 2016-12-13T19:40:34Z

@Juanvulcano what's the story with this block of code?

+                for i in range(3):
+                    client = get_client("API")

I don't understand why one would do this in a loop...

Juanvulcano mentioned this pull request Dec 7, 2016

populate_db should generate some default presence data #2426

Closed

Juanvulcano force-pushed the database branch from 9efdb1e to 05e7878 Compare December 7, 2016 01:41

umairwaheed reviewed Dec 7, 2016

View reviewed changes

Juanvulcano force-pushed the database branch from 05e7878 to 9bce672 Compare December 7, 2016 20:52

Juanvulcano force-pushed the database branch 2 times, most recently from 9a8d39f to 94d4baf Compare December 7, 2016 22:29

Juanvulcano force-pushed the database branch 2 times, most recently from b894637 to f1d2f41 Compare December 8, 2016 00:59

Juanvulcano force-pushed the database branch 11 times, most recently from a988318 to ec94629 Compare December 8, 2016 21:31

Juanvulcano force-pushed the database branch from ec94629 to 3e3186f Compare December 8, 2016 22:01

Juanvulcano force-pushed the database branch from 3e3186f to ce4f8d4 Compare December 8, 2016 22:22

Juanvulcano force-pushed the database branch from ce4f8d4 to 4831132 Compare December 8, 2016 22:32

Juanvulcano force-pushed the database branch 4 times, most recently from 7313b77 to 9fec7ba Compare December 8, 2016 23:06

Juanvulcano changed the title ~~Added UserPresence to population script~~ Fixes: #2426 Populate with UserPresences Dec 8, 2016

umairwaheed reviewed Dec 9, 2016

View reviewed changes

Juanvulcano force-pushed the database branch from 9fec7ba to dce3cba Compare December 9, 2016 19:20

Juanvulcano force-pushed the database branch from dce3cba to fa9e65f Compare December 9, 2016 19:50

showell reviewed Dec 9, 2016

View reviewed changes

Juanvulcano force-pushed the database branch from fa9e65f to 61e9a16 Compare December 9, 2016 20:18

Add UserPresence rows to test databases.

af3a785

Fixes: zulip#2426 mypy: Added annotations to populate_db.py

Juanvulcano force-pushed the database branch from 61e9a16 to af3a785 Compare December 9, 2016 20:50

showell merged commit 9869fb8 into zulip:master Dec 9, 2016

timabbott mentioned this pull request Dec 16, 2016

Naive datetime warnings in populate_db #2729

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes: #2426 Populate with UserPresences #2626

Fixes: #2426 Populate with UserPresences #2626

Juanvulcano commented Dec 7, 2016 •

edited

smarx commented Dec 7, 2016

umairwaheed Dec 7, 2016

umairwaheed Dec 7, 2016

umairwaheed Dec 7, 2016

umairwaheed Dec 7, 2016

umairwaheed commented Dec 7, 2016

Juanvulcano commented Dec 7, 2016

Juanvulcano commented Dec 7, 2016

mjec commented Dec 8, 2016

Juanvulcano commented Dec 8, 2016 •

edited

timabbott commented Dec 8, 2016 •

edited

timabbott commented Dec 8, 2016

Juanvulcano commented Dec 8, 2016 •

edited

timabbott commented Dec 8, 2016

timabbott commented Dec 8, 2016

Juanvulcano commented Dec 8, 2016 •

edited

umairwaheed Dec 9, 2016

umairwaheed Dec 9, 2016

umairwaheed commented Dec 9, 2016

showell commented Dec 9, 2016

showell Dec 9, 2016

showell commented Dec 9, 2016

timabbott commented Dec 13, 2016

Fixes: #2426 Populate with UserPresences #2626

Fixes: #2426 Populate with UserPresences #2626

Conversation

Juanvulcano commented Dec 7, 2016 • edited

smarx commented Dec 7, 2016

umairwaheed Dec 7, 2016

Choose a reason for hiding this comment

umairwaheed Dec 7, 2016

Choose a reason for hiding this comment

umairwaheed Dec 7, 2016

Choose a reason for hiding this comment

umairwaheed Dec 7, 2016

Choose a reason for hiding this comment

umairwaheed commented Dec 7, 2016

Juanvulcano commented Dec 7, 2016

Juanvulcano commented Dec 7, 2016

mjec commented Dec 8, 2016

Juanvulcano commented Dec 8, 2016 • edited

timabbott commented Dec 8, 2016 • edited

timabbott commented Dec 8, 2016

Juanvulcano commented Dec 8, 2016 • edited

timabbott commented Dec 8, 2016

timabbott commented Dec 8, 2016

Juanvulcano commented Dec 8, 2016 • edited

umairwaheed Dec 9, 2016

Choose a reason for hiding this comment

umairwaheed Dec 9, 2016

Choose a reason for hiding this comment

umairwaheed commented Dec 9, 2016

showell commented Dec 9, 2016

showell Dec 9, 2016

Choose a reason for hiding this comment

showell commented Dec 9, 2016

timabbott commented Dec 13, 2016

Juanvulcano commented Dec 7, 2016 •

edited

Juanvulcano commented Dec 8, 2016 •

edited

timabbott commented Dec 8, 2016 •

edited

Juanvulcano commented Dec 8, 2016 •

edited

Juanvulcano commented Dec 8, 2016 •

edited