Raw data

[mx03]

This repo looks very interesting. Is there a possibility getting the raw data before merge to optimize the merge between the anime sites?

[manami-project]

Hi @mx03,
I don't really see any possibility to provide the raw data, because the raw data is huge.

I admire that you would like to improve the merging. But I would love to point out a few things:

First I would say that it is not possible to achieve a 100% match with an automated, leightweight process. Merging a data set of 52776 into 29702 entries is pretty okay, I guess. From the logs I can say that the logic itself seems to be pretty good. A Quick look at the log files revealed that in 7257 cases a merge has been actively prevented, because properties differed. Of course there are cases in which an anime exists only on solely one of the anime database sites and so on.

Secondly you have to make up your mind about your priority. My priority here is to make correct merges instead of having a high density. Hence the actively prevented merges I mentioned above. I rather not merge an entry than merging it with an entry that is not really the same production. That happened in the past and that was the point I made that decision.
The point is that I don't lose any data except for the link between two anime database sites indicating the same anime. But that also depends on what I do with the data. If I search an entry based on it's name, for example, there is a possibility that I can fetch both source links and therefore create a merge by making a search query on the title.

[mx03]

I can understand your concern, correct merges are most important thing with a database like yours.

I was looking for a database like yours to prevent the vendor lock-in of my data. But two test picks "Death Note" and "Cowboy Bebop" are cut in 2 datasets ( I think because of the active merge prevention). So my thought better automatic merge or force datasets by hand together.

[manami-project]

I can totally relate. I started with MAL and added more, because I wanted to prevent a vendor lock-in for myself as well. Or at least I wanted to be able to shift smoothly in a short period of time. And of course, because I though others might appreciate it ;)

I personally think doing additional merging manually is a crazy amount of work.

Back to your test picks:
Both Death Note and Cowboy Bebop went into additional properties check and got active merge prevention, just as you assumed.
So i checked the log files again and in 6807 cases the duration was different. For both your test picks that was the case as well. So the question here is: Is it possible to increase the merge density without losing quality by making the check on the duration more fuzzy?

[mx03]

Am I'm wrong or is the duration of an standard tv anime show on anidb always 25min?

In my option is the duration a bad factor, because most tv anime series has 23-25min. The value is for the type "tv shows" more an issue, but for web/ona, ova, movie its should be a required check.

[manami-project]

I'm not sure about anidb. I would need to check that specifically.
I know that the others vary a lot though. I made an analysis on the difference in the duration and I decided that I will start making the duration check a little bit more fuzzy with the next update.