Migration from third services - Framabag

[nicosomb]

Currently, Framabag can't import datas from other services.

We have to change the import system to upload a file and import items 10 per 10.

[nicosomb]

See #365.

[nicosomb]

@mariroz: interested with this issue?

[mariroz]

yes, this is, of course, important task, and can work on it, but only if it's not urgent.
I will be totally unable to work on wallabag during next 5-6 days as going to spent some time on skiing. I hope, will read comments, however.
Also, here is small list of tasks, I plan to do, if noone will do it in meantime:

• wallabag a link in wallabag wallabag a link in wallabag #505
• List of existing Tags List of existing Tags #477

both are simple and can be done quickly.
Then, if you think it will be useful, I would like to make some refactoring related to check_setup.php: maybe it will be better to avoid running it each time script is started, I think, this can be done more effective using try/catch construction.
Then current issue #426, and then, maybe, will try to work on handling of txt, pdf video etc (#508, #444,...) as this is interesting task.
But all this after 5-6 days of silence.

[nicosomb]

no problem, thank you so much for your help and good holidays ;-)

[nicosomb]

I started this feature on upload-file branch (https://github.com/wallabag/wallabag/tree/upload-file).

• upload form to upload a file into cache folder
• when we click on "Import from Pocket / etc.", new entries are added in database, without title / content, to avoid max execution time with big files.
• cron system (or via a link) to fetch content for X entries.

All seems to be good but cron crashes, due to getPageContent() (I moved it into Tools.class.php to be accessible from cron.php): content of first fetched article is displayed. I think it's due to ob_start.

@mariroz: do you have an idea?

[mariroz]

will try to investigete this evening.

[nicosomb]

@mariroz I think we have to delete pocket/readability/instapaper export file when import is over. What do you think?

[mariroz]

@nicosomb , of course, all parsed import files should be deleted.
Also, maybe there is no sense to store it at all: every import file can be parsed w/o moving it to cache folder. Pls also note, that every outside file inside application web root is potential security problem.

[nicosomb]

I store this file in cache to separate import and fetch content actions.
Like this, we can fetch content for 10 articles, then 10 articles, etc.

When you have 1000 links to import, you avoid timeout.

[mariroz]

yes, you fetch content for 10 articles at once.
BUT, as I can see in code, you insert all entries into DB at once with empty content. So, after this step you don't need file any more. Pls correct me, if I'm wrong.

[nicosomb]

You're right. We can delete this file.

[mariroz]

Also, sorry to be so critical, but I don't like idea with cron: it makes application essentially more complicated for common users. Pls don't forget, that php cli is often not available on common hosting plans (the same about cron - it may be inavailable, it is complicated to configure, use and debug in case of problems for usual user).
I think, that it will be much better to make import using series of reload (or ajax requests) of the same import page with some progress message. For example, if user press "import", he will be redirected to import page, where he will see something like "importing articles 1-10", then redirect in 5 sec to the same page but with message "importing articles 10-20" etc. Or don't redirect but request ajax action to do the same. Hope, you understand, what I mean.

[nicosomb]

Import file is now deleted when import is over (see 8d7cd2c).

it makes application essentially more complicated for common users. Pls don't forget, that php cli is often not available on common hosting plans

I know this and that's why we can click on a link to fetch content for 10 articles.
We can improve this system with ajax, in next version.

What do you think about that, @mariroz?

[nicosomb]

This feature was added on 1.6. I close this issue.