Search stopped indexing and I don't know why

[solarbirdy]

Hi! Running 4.2.1 on debian, up to date in general.

Full-text search has been working great and is still working - but it stopped adding anything to the search database late on October 28th or very early October 29th. Nothing's in syslog, elasticsearch (v 7.17.15, pulled from https://artifacts.elastic.co/packages/7.x/apt stable InRelease) says it's happy and running fine, there's nothing in syslog, last entry in elasticsearch.log is that cluster health status is green (after a restart), but I am seeing a whole lot of this in sidekiq:

Chewy::ImportFailed: Import failed for TagsIndex with: Index errors: `{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [last_status_at] of type [date] in document with

Anybody know what might be up with this? I've done a lot of searching without finding anything so...

[solarbirdy]

okay I did more digging and I'm seeing this:

{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [last_status_at] of type [date] in document with id '16013'. Preview of field's value: '11968-11-07T17:22:13.599Z'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [11968-11-07T17:22:13.599Z] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}

the document IDs change but the time field doesn't, and that's certainly an interesting time and I have no idea what to do about that so... ideas?

[solarbirdy]

Michael asked to open an issue so I did:

https://mastodon.murkworks.net/@moira/111432882956209704

Issue is here:

#27970

But now it's clearly a duplicate of this:

#27639

But I don't know how to find and remove the posts with the invalid date. Help?

[vmstan]

You can fix this directly by correcting the date on the bad post.

From your Mastodon server, in the mastodon directory, open a Rails Console:
bundle exec rails console
To find the status(es):
Status.where(created_at: "11968-11-07T17:22:13.599Z")
To correct the date:
Status.where(created_at: "11968-11-07T17:22:13.599Z").update_all(created_at: "2023-10-26T03:39:25.142Z")

For the tags:
To find them:
Tag.where(last_status_at: "11968-11-07T17:22:13.599Z")
To update them:
Tag.where(last_status_at: "11968-11-07T17:22:13.599Z").update_all(last_status_at: "2023-10-26T03:39:25.142Z")

Credit to @rrgeorge on these instructions.

[solarbirdy]

thank you very much!

in case it's of interest this is the offending post:

[#<Status:0x00007f6c3c4c47f8
id: 111313272683052719,
uri: "https://mk.itzti.li/notes/9la79fmeadxw63t4",
text:
"

📌 Hilo fijado: Recomendaciones musicales chidas

Aquí iré poniendo música para licuarse el cerebro, elevar el espíritu, levantar la moral o desprenderse del cuerpo, todo desde mi biblioteca, quizá usando <a href="https://mk.itzti.li/tags/Ampache" rel="tag">#Ampache o, como siempre, <a href="https://mk.itzti.li/tags/Invidious​" rel="tag">#Invidious​ (Ahora también <a href="https://mk.itzti.li/tags/Piped" rel="tag">#Piped)

Así para encontrarlos mejor, porque luego se me pierden, jaja
<a href="https://mk.itzti.li/tags/Recomendación" rel="tag">#Recomendación <a href="https://mk.itzti.li/tags/Musical" rel="tag">#Musical
<a href="https://mk.itzti.li/tags/Música" rel="tag">#Música <a href="https://mk.itzti.li/tags/Music" rel="tag">#Music <a href="https://mk.itzti.li/tags/FediMusic" rel="tag">#FediMusic <a href="https://mk.itzti.li/tags/Invidious" rel="tag">#Invidious
<a href="https://mk.itzti.li/tags/Flan" rel="tag">#Flan <a href="https://mk.itzti.li/tags/Custard" rel="tag">#Custard <a href="https://mk.itzti.li/tags/CusTom" rel="tag">#CusTom

",
created_at: Thu, 07 Nov 11968 17:22:13.599000000 UTC +00:00,
updated_at: Sat, 28 Oct 2023 15:09:07.728736000 UTC +00:00,
in_reply_to_id: nil,
reblog_of_id: nil,
url: "https://mk.itzti.li/notes/9la79fmeadxw63t4",
sensitive: false,
visibility: "public",
spoiler_text: "",
reply: false,
language: nil,
conversation_id: 8112423,
local: false,
account_id: 111313267981205598,
application_id: nil,
in_reply_to_account_id: nil,
poll_id: nil,
deleted_at: nil,
edited_at: nil,
trendable: nil,
ordered_media_attachment_ids: nil>]

[rrgeorge]

I am fairly certain that is the same offending post the caused the issue for me.

[vmstan]

Totally is. I copied your commands from Discord 😆