-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request]: CSV import/export #64
Comments
I agree with needing import/export but am not sure about the export format being CSV since it does not support nested data structures. I think JSON would be better. |
I think CSV is the easiest and most universal but yes, if some functionality actually does require nested data structures, then creating a JSON file from a CSV isn't all that complicated anyway, so either will work. What use case do you see for nested data structures? Both Goodreads and Simkl offer CSV exports (which makes CSV kind of a universal import format as well), so at least for media tracking there doesn't seem to be a need for anything more complicated. |
For example a media item can be added to the seen history multiple times. How can I represent this in a CSV? |
I think Simkl "solves" this by only saving the latest view date in the CSV, plus potentially a view count. But you are correct, the view history (dates) for the other times would be lost in that case, while they could be preserved within a JSON structure. Simkl also offers JSON export functionality which preserves all information. It's certainly a tradeoff, complete information vs simplicity of the format. Since either one is text based and it's easy enough to convert one to the other manually if necessary, I think either one would be great options. |
Cool. I think I'll go with JSON exports then. |
Awesome :) Imports as well? That's probably the harder part to get right (name matching and reports on what got matched), although personally I think it's the more important one. With a huge existing collection, entering data manually might not be feasible at all, which means switching to Ryot might be impossible in that case. |
The matching idea is pretty good and I'd like to implement it for Goodreads too. I have an idea for an import view in mind too. I'll draw it out in Excalidraw and upload it here later. |
Nice, I really like Excalidraw! It probably needs either more columns for more types of IDs (IGDB for example) or maybe a column "ID type" and another "ID" so that any type of ID can be handled in a flexible manner, so the format doesn't need to change when new types of tracking are added? Or maybe some type of linked tables since one item should probably be able to have multiple IDs (e.g. both a IMDB and TVDB ID). I like the "is perfect match" part. Maybe instead of having it boolean, make it "match type" with 3 options, "perfect", "best effort", "none". For practical usability, it would also be awesome to have the list easily copy&pastable so that the "best effort" ones can be checked manually and the "none" ones can be added manually. The ideal most awesome and best solution would of course be a kind of dropdown menu that shows all matches (for the fuzzy name match) so if the ideal one wasn't found, the correct one can be selected manually with the dropdown menu right in the import section. But that's probably too complicated to offer a UI for that but being able to copy&paste the names into some spreadsheet to go through them manually should be enough. The way Simkl handles it is that after the import, it saves a CSV file with all errors, which then is a great reference for the manual checks / corrections. Not sure whether that is feasible for Ryot, though. I also just checked the Simkl export. They use the following columns:
|
Would it be possible for you to share this SIMkl export with me? My email ID is in my github profile. |
Sure, no problem at all. I assume you don't need my full history for everything, a few lines for TV shows, movies and anime (that's what Simkl tracks, sadly no games) are enough, right? I trimmed it down to examples which are representative I think, this easily fits into a comment here:
While their export is only free for regular users, their import is free right away for everyone. The CSV import (which is the best I've found for any media tracking tool so far, probably a good role model even for a JSON import) is here: https://simkl.com/apps/import/csv/ |
It would be great if you could share the entire export so that i could test this feature to the fullest. |
@Mega-Volti should the json export be a file or an endpoint? |
Personally, I'd be perfectly happy with a file. The main use for the export would be backup/storage of data, so being able to save a file and put that somewhere safe is the main use case I had in mind. |
I think I'll go with an endpoint since the response could be written to a file if needed and also be used for live data imports somewhere else. |
@Mega-Volti I have decided to go with a json import to allow for nested data. I have come up with the following import format: interface Review {
text?: string;
/** Score out of 100 */
rating?: number;
spoiler?: boolean
}
interface Seen {
ended_on?: Date;
/** if a show, which season was seen? */
show_season_number?: number;
/** if a show, which episode was seen? */
show_episode_number?: number;
/** if a podcast, which episode was seen? */
podcast_episode_number?: number;
}
export interface ImportItem {
source_id: number; // In this case, this will just be the serial number (index in array)
source: 'anilist' | 'audible' | 'google_books' | 'igdb' | 'itunes' | 'listennotes' | 'openlibrary' | 'tmdb';
identifier: string;
lot: 'audio_book' | 'anime' | 'book' | 'podcast' | 'manga' | 'movie' | 'show' | 'video_game';
collections: string[];
reviews: Review[];
seen_history: Seen[];
} Does this look good to you? Reason I want to go with structure is because Ryot already knows how to import this format: ryot/apps/backend/src/importer/mod.rs Lines 107 to 116 in 34012fe
|
That does seem reasonable, yes. It also looks like a CSV could be converted into this JSON structure quite easily. It might even be possible to offer both imports, by simply having a small CSV->JSON converter run beforehand? |
I don't think I will provide that since it is dead easy to convert b/w those two formats. It would be better if we create a discussion with those scripts. |
@Mega-Volti I did not include a dedicated UI since it would be too big a feature. If the definition of the expected types are not enough, you can read the comments here for full explanations: ryot/apps/backend/src/models.rs Lines 517 to 572 in 1dea787
Let me know how your importing goes when you're done! |
Good point, I'll be looking forward to having a nice script to do that, manually converting my giant CSV into a JSON file would be quite a hassle :) I tried testing the import with sample video games. To see an example of the format, I added a test game ("God of War (2018)" in my case) within Ryot and exported it, to see the exported JSON. I then wanted to just copy that and add my games-to-import. This is where I hit the first roadblock: I only see IDs there, no field for the game name. The point of importing would be to import with game names, I don't have IGDB IDs for all these games. How exactly would an import JSON be formatted in order to allow importing a list of games, of which I have the name as wel as the completion date, but nothing else? |
@Mega-Volti Of course if this was possible on backend it would be better , but then resolution of conflicts isn't straightforward |
Right, having that functionality built in would be amazing. Exactly as you describe it, if there is a perfect match then display it, otherwise display the cloesest match with some sort of marker that is was only a close match, and enable picking the game from the regular search UI in all other cases. How did you match your shows to get all the IDs? My list is very, very long, doing that manually is not feasible, and I don't know of any UI that does it for game names. And without being able to provide game IDs, I won't be able to use the importer properly, right? |
The UI you are describing would be pretty complicated and as such would take me a lot of time to build; something I am loath to do since this will not be a highly used feature. I can think of two solutions:
|
@Mega-Volti When I say search - I searched the Ryot graphql API, but I suppose its probably a very rough idea and could instead be done at source IGDB/TMDB and etc |
For general import/export functionalities, CSV is the most flexible. Ultimately, most data from other services can be converted to CSV format (more of less easily), making it a near universal import/export format.
CSV import is the harder of the two I assume. Features ideal for this are:
As example, I think Simkl (https://simkl.com/) does this nearly perfect. It does name matching, presents a tabular-style view for easy manual checking of important items and it creates a CSV error report for items that couldn't be matched which enables reasonably easy manual import of the rest.
CSV export on the other hand should be reasonably easy I think. That should just be a dump of all relevant fields in the database, written to a text file, using the same column headers as the import functionality. Both Simkl and Goodreads have excellent CSV export functionality which might be a good template.
The text was updated successfully, but these errors were encountered: