Search produces no results

[stone212]

After upgrading to Ubuntu 18.04 and going through many issues including #12012 and #12009 search does not give the old error. But it does not show results.

This applies to all streams and all realms on this zulip server. Result of any search is:

No search results

You searched for: time

OR

Nothing's been sent here yet!

Why not start the conversation?

This is when searching in the top bar above the conversation panel for words that are definitely in the conversation.

/var/log/zulip/errors.log has no entry on these searches

[stone212]

@timabbott

This Issue continues after successful upgrade-zulip-from git master.

No errors in errors.log however I search for the word aristotelian and then to grep -nr 'aristotelian' in /var/log/zuliup and this is displayed:

django.log:20:[pid: 7814|app: 0|req: 366/1089] 5.45.179.182 () {50 vars in 1425 bytes} [Thu Apr 25 01:57:51 2019] GET /json/messages?anchor=-1&num_before=50&num_after=50&narrow=%5B%7B%22negated%22%3Afalse%2C%22operator%22%3A%22search%22%2C%22operand%22%3A%22aristotelian%22%7D%5D&use_first_unread_anchor=true&client_gravatar=true => generated 156 bytes in 21 msecs (HTTP/1.1 200) 7 headers in 208 bytes (1 switches on core 0)

The display that corresponds to that entry is:

No search results

You searched for: aristotelian

[stone212]

This is an unresolved issue but I am using full text search now and can not help de-bug. https://zulip.readthedocs.io/en/latest/subsystems/full-text-search.html

[timabbott]

@stone212 did you switch to pgroonga, or not? I will assume that you are not using pgroonga as your search backend in the below, please switch back if you are, since I don't want that to be a variable in this investigation.

Make sure that (1) the process_fts_updates job is running in supervisorctl status -- it is what updates the search index. You likely will need to restart that job given the fiddling you did in previous issues. Whether that job is working will impact whether searching for a word you sent in a message 10 seconds ago works. For older messages, I suppose it's possible you need to re-backfill your message history. The following SQL command, run in a manage.py dbshell, should do that:

UPDATE zerver_message SET search_tsvector = to_tsvector('zulip.english_us_search', subject || rendered_content)

[zulipbot]

Hello @zulip/server-search members, this issue was labeled with the "area: search" label, so you may want to check it out!

[timabbott]

@stone212 have you had a chance to try that?

[stone212]

@timabbott

did you switch to pgroonga, or not?

Yes and the system is working. Since it is sometimes 18 days between replies from you, and since you often reply without reading what I wrote and because this is a production system with many users - having a stable system after all of these weeks is the priority. That is why I said I can not help you de-bug this Issue at this point.

I intend to migrate to a new server soon so working on migration issues will be my priority after I catch upon the 150 other servers which took a lower priority because of these zulip issues. Hopefully the migration wil be smoother than it went the first time but if not I will open new Issues.

[stone212]

@timabbott Anyway why did you suggest pgroonga if you didn't want me to use it (#12206)?

And where are the instructions to "switch back"? You skipped over that part in your comment above even if it was prudent for other reasons. It's these half answers and disconnected responses that are making the original problem (which was caused by poor Zulip docs) take more than a month to fix.

[timabbott]

@timabbott Anyway why did you suggest pgroonga if you didn't want me to use it (#12206)?

PGroonga is totally fine to use (actually better if you have users who use multiple languages), there are several production sites happily using it and there's a good chance we'll migrate everyone over to it. It's just a less mature implementation. We just can't debug your problem with the other backend without switching back.

I don't think it's a priority for the project to debug the search issue you ran into, because I don't expect that other folks will end up in the same situation. So if you're happy with the PGroonga search experience, I'm happy to close this issue.

And where are the instructions to "switch back"? You skipped over that part in your comment above even if it was prudent for other reasons.

The instructions for switching back are documented at the bottom of the article on PGroonga.
See https://zulip.readthedocs.io/en/latest/subsystems/full-text-search.html#how-to-disable-full-text-search-across-all-languages.

It's these half answers and disconnected responses that are making the original problem (which was caused by poor Zulip docs) take more than a month to fix.

@stone212 I'm sorry you had extended downtime here, because it sucks, but blaming Zulip and me for your downtime is unfair.

• Both Ubuntu and Zulip have documentation recommending against doing a do-release-upgrade on a running production system and that you should take a backup first (I think Ubuntu even warns you when you run the command). We recommended against doing so on a production precisely because it wasn't at the time something we had documentation for (I added this documentation a few weeks ago). Your choice to do that do-release-upgrade without providing us with what precisely you ran is what got you into a broken situation that we could not reproduce.
• Your organization doesn't pay for a formal support contract, which is you do if you want someone to help them through tricky technical issues with an SLA. I know your organization has donated in the past, which we very much appreciate, but that's different from paying a negotiated rate for a specific level of technical support service.
• The very first advice I gave you a month ago for how to fix full-text search, with the CREATE EXTENSION ... command in a postgres shell, followed by supervisorctl restart all, should have been all you needed to do to eliminate that first error, and it appears you failed to actually run it. Inhttps://github.com/Search (only) produces "Unable to connect to Zulip. Updates may be delayed. Retrying soon... Try now." #12009#issuecomment-480559204, you didn't include a ; at the end of the SQL statement, and thus didn't actually run anything in the SQL shell. While perhaps I should not have assumed that you knew SQL statements need to end with a ; and spelled out the command precisely, you definitely should have mentioned that the command hung and never completed! Then it would have been obvious that it didn't do anything and we'd have corrected it right then. Neglecting to mention that is the root cause of your being stuck with search broken for weeks, because in order to help further, I had to first try to reproduce the weird situation (which was impossible) and then think about explanations for it not fixing your problem like the CREATE EXTENSION command creating the extension in the wrong postgres database/schema/etc.

[kou]

I'm the PGroonga developer. I'll be able to help you.

I'll try upgrading my Zulip production server to 18.04 from 16.04 in a few days. The server is using PGroonga.

[stone212]

@kou Thank you but pgroonga is already working very well. I do have concerns about migrating to another server with an original using pgroonga. Do you have any documentation for this?

[stone212]

@timabbott

I'm sorry you had extended downtime here, because it sucks, but blaming Zulip and me for your downtime is unfair.

No, Tim, it's not unfair. Your docs sucked and your replies (or lack of replies) sucked more. Because of the work I have put in, your docs are much better (you're welcome). But because you want to call me out here I'll go through your points:

Both Ubuntu and Zulip have documentation recommending against doing a do-release-upgrade on a running production system and that you should take a backup first (I think Ubuntu even warns you when you run the command).

I did have a backup. I have had a backup this whole time. I told you that.

But anyway your backup/restore process is not working like the docs describe so it doesn't matter does it?

Your choice to do that do-release-upgrade without providing us with what precisely you ran is what got you into a broken situation that we could not reproduce.

No it isn't. The system was broken before I posted. That is why I posted. And I have answered every question you asked. If you wanted more information you could have asked more questions. But you didn't. Instead you disappeared when I tried the things you suggested. Sometimes you suggested the same thing right after I told you I just did it. You weren't reading my replies Tim.

Your organization doesn't pay for a formal support contract,

I don't have an organziation and anyway I am posting bugs with Zulip and with the zulip docs. Again I say, you're welcome.

The very first advice I gave you a month ago

The very first advice was more than a month ago, and I followed it, and it wasn't what you just said it was. There was other advice you did give me a month ago which I followed as you explained, and I posted the exact commands I ran for you to look at. I asked you (twice!) to verify that I did what you wanted. You ignored this both times. Then (a month later) you mentioned something else that helped shed light on this - although it still was not a solution to the overall problem which is why I opened this Issue.

you definitely should have mentioned that the command hung and never completed! Then it would have been obvious that it didn't do anything and we'd have corrected it right then.

I did tell you that, two times. You did not reply. That's why I'm so pissed off and why I am right to be. 18 days man.

No one expects perfection, no one expects service for an open-source project. But everyone who is an active member of the community who takes time to write full bug reports and to read the docs and to make suggestions to make zulip better should expect the dev team's participation in fixing bugs that were created by their own poor docs (proven because you now have good docs based on my experiences).

This is not a world ender. I am still participating in the Zulip community, I still want to help make Zulip better by making good bug reports and suggestions. I consider this a mostly friendly disagreement. But I am not going to roll over when you deflect blame onto me for (an honest set of) mistakes that you made, without me pointing out your error.

You have a good product overall. It will be very nice to get it working again.

[kou]

Oh, sorry. I misunderstood.

Normally, you can migrate your server by the followings:

1. Install fresh Zulip to new server https://zulip.readthedocs.io/en/latest/production/install.html with the same /etc/zulip/settings.py content with old server.
2. Backup data in old server: https://zulip.readthedocs.io/en/latest/production/maintain-secure-upgrade.html#backups
3. Restore backup date in new server: https://zulip.readthedocs.io/en/latest/production/maintain-secure-upgrade.html#restoring-backups

I'm not sure this is documented. Sorry.

BTW, I've upgraded my production Zulip server to 18.04 from 16.04. And I found some problems. I've created a pull request to fix this as #12261.

[kou]

Ah, sorry.

1. Install fresh Zulip to new server https://zulip.readthedocs.io/en/latest/production/install.html with the same `/etc/zulip/settings.py` content with old server.

This is needless. Because

3. Restore backup date in new server: https://zulip.readthedocs.io/en/latest/production/maintain-secure-upgrade.html#restoring-backups

already says

First, install a new Zulip server through Step 3 with the version of both the base OS and Zulip from your previous installation.

[stone212]

@kou Are you replying to me? Migrating to a new server is a different Issue. This is about Search producing no results after upgrading to Ubuntu 18.04 and following the steps @timabbott explained in 12009.

[kou]

Yes. I answered your question at #12206 (comment)

I do have concerns about migrating to another server with an original using pgroonga. Do you have any documentation for this?

Did my answer solve your concern?

[stone212]

@kou I don't know what you're talking about but your comment does not answer this Issue and it does not give new information about migrating to a new Zulip server but anyway that is not this Issue. Please don't confuse this Issue by giving details about a different subject.

[kou]

What is the remained problem in this issue?
Can we close this issue? Or should we resolve something?

[stone212]

@kou

The original Issue was not resolved was it? Please see my original comment. I am un-subscribing from this thread because I am using pgroonga and can not help test.

[timabbott]

@kou the main discussion on this thread isn't related to pgroonga, but the tsearch_extras search backend. This issue is about recovering from having done a pg_upgradecluster without the appropriate full-text search backend enabled on the new postgres version. Thanks for testing the pgroonga postgres OS version upgrade path; I hadn't thought to test that when writing the instructions.

This is not a world ender. I am still participating in the Zulip community, I still want to help make Zulip better by making good bug reports and suggestions. I consider this a mostly friendly disagreement.

@stone212 agreed. I agree I could have noticed the missing ;.

You did not reply. That's why I'm so pissed off and why I am right to be. 18 days man.

Yes, and I apologize for this, as it is not the norm for getting help on GitHub issues in this project (as I think you know, most issue thread comments get a reply to me within a day in normal circumstances). I probably should have mentioned this earlier in the thread, but I was very ill with a flu for like 3 weeks covering this interval, which is why my replies were very slow for this issue and often brief. So I was doing the best I could to help with my limited useful time available for this and other reports.

As we don't have a way to debug it, I'm going to close this issue; the thread will be useful if someone else ends up in this situation, but we don't have a reproducible problem for new users, and in any case, arguably the fix to this problem for future users is the new documentation we wrote on how to migrate the Zulip database to a new Ubuntu version without getting into this situation. I'm happy to reopen (or in any case investigate further) if there's an opportunity to do so, since I am curious what the state is, though it'd probably be a lot easier to do interactively on chat.zulip.org as I'll likely want to try like 10 different things in quick succession.

@stone212 thanks for all the debugging back and forth, and for providing the impetus for us to have good documentation for the do-release-upgrade process! I'm sure other users will benefit from this a lot in the future.

[stone212]

Thanks for testing the pgroonga postgres OS version upgrade path; I hadn't thought to test that when writing the instructions.

Yes! This might help my migration issues because I use pgroonga so this thread may have helped. But it is correct to close it since we can not re-produce.

[ATHENA12D]

Hi, i have same this issue: i deployed the zulip server in docker and use a remote postgresql, when i restore the backup of db from the old serevr to the new remote db, i faced an issue with searching, when i search on any thing there is no result, but when i search on number like "360" is search is working with numbers, i can't understand where is the issue?