How to delete a spider and the data?

[phongtnit]

Hi,

I want to delete immediately all data of a spider. I executed the following steps instead of the guide http://docs.pyspider.org/en/latest/Frequently-Asked-Questions/ (I don't want to wait 24 hours as the tutorial does):

1. I remove spider's document in projectdb database.
2. I remove spider's collection in taskdb database.
3. I remove spider's collection in resultdb database.

However, when I recreate the spider with the same name. I checked http://x.x.x.x:5000/tasks?project=spidername. I saw old data of the spider. Why?

Please guide me all steps to delete immediately all data of a spider. Thanks in advance,

[binux]

http://docs.pyspider.org/en/latest/Frequently-Asked-Questions/#how-to-delete-a-project
you can change scheduler.DELETE_TIME by scheduler argument --delete-time

[phongtnit]

Hello,

Do you mention that I will stop the schedule and restart with --delete-time option? For example, pyspider -c config.json scheduler --delete-time=1 ( 1 = an hour)

Thanks for your support,

[binux]

--delete-time=1 means 1 second, you could put argument 'delete_time' to config.json as well.

[phongtnit]

Hello binux,

I did the following steps:

1. Add delete_time option to config.json file

{
  "taskdb": "mongodb+taskdb://127.0.0.1:27017/pyspider_taskdb",
  "projectdb": "mongodb+projectdb://127.0.0.1:27017/pyspider_projectdb",
  "resultdb": "mongodb+resultdb://127.0.0.1:27017/pyspider_resultdb",
  "message_queue": "redis://127.0.0.1:6379/db",
  "webui": {
    "username": "admin",
    "password": "xxxxxx",
    "need-auth": true
  },
  "delete_time": 60
}

2. Setup delete a project on webui, please refer http://i.imgur.com/mfmh9WU.png
3. Kill the scheduler process
4. Start scheduler process without deleting data folder of the scheduler.

nohup pyspider -c config.json scheduler >> logs/scheduler.log 2>&1&

5. Wait more than 20 minutes
6. I cheked webui, the project didn't delete. Also, I checked documents of the project in resultdb database, their documents didn't remove too.

What should I do? I'm using pyspider 0.3.6

Thanks for your help,

[binux]

{
  "taskdb": "mongodb+taskdb://127.0.0.1:27017/pyspider_taskdb",
  "projectdb": "mongodb+projectdb://127.0.0.1:27017/pyspider_projectdb",
  "resultdb": "mongodb+resultdb://127.0.0.1:27017/pyspider_resultdb",
  "message_queue": "redis://127.0.0.1:6379/db",
  "webui": {
    "username": "admin",
    "password": "xxxxxx",
    "need-auth": true
  },
  "scheduler": {
    "delete_time": 60
  }
}

[phongtnit]

Hello binux,

That's works now thanks to your setting.

Many thanks for your script and awesome help,

[phongtnit]

Hi binux,

After successful deleting a project, I added the project again with the same name. It showed that pyspider keep old data of the process of the spider. Please refer http://i.imgur.com/Maqhh7I.png

I guess this is a small bug.

[visoar]

After successful deleting a project, I added the project again with the same name. It showed that pyspider keep old data of the process of the spider. Please refer http://i.imgur.com/Maqhh7I.png

have the same problem.

[binux]

it should be fixed via 468202f