Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to update the database #19

Closed
1223334444abc opened this issue Jul 10, 2023 · 11 comments
Closed

How to update the database #19

1223334444abc opened this issue Jul 10, 2023 · 11 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@1223334444abc
Copy link

I tried:

date-after: 20230531
update-after: 20230531

Whether they are written in the configuration file simultaneously or separately, new posts from June and July cannot be downloaded.

I hope to have incremental updates to the database: download new posts and update posts with changes. How should I write the configuration file?

Are there any conflicting relationships between different settings? A clearer explanation is indeed needed.
Does "-- date" represent the upload date of the post on the source website, and "-- update" represent the modification date of the post on Kemono? Can these two settings be used simultaneously and what specific effects will they have?

@elvis972602
Copy link
Owner

I apologize, I seem to have set the wrong variable name...
And I will add a detailed tutorial document soon.

@elvis972602 elvis972602 added the bug Something isn't working label Jul 10, 2023
@1223334444abc
Copy link
Author

1223334444abc commented Jul 10, 2023

fetching post list page 0...
fetch post list error: Get "https://kemono.party/api/fanbox/user/*****?o=0": EOF
fetch post list error: Get "https://kemono.party/api/fanbox/user/*****?o=0": EOF
fetch post list error: Get "https://kemono.party/api/fanbox/user/*****?o=0": EOF
panic: fetch post list error: maximum retry count exceeded

goroutine 1 [running]:
github.com/elvis972602/kemono-scraper/kemono.(*Kemono).Start(0xc00033c3f0)
C:/Users/elvis/GolandProjects/Kemono-scraper/kemono/kemono.go:234 +0x5d7
main.main()
C:/Users/elvis/GolandProjects/Kemono-scraper/main/main.go:470 +0x3ae5

There is another bug here. When an error occurs while obtaining the list, the program will directly interrupt. I seem unable to set the number of retries here, so I can only run the entire program again and completely.

@elvis972602 elvis972602 added the enhancement New feature or request label Jul 10, 2023
@elvis972602
Copy link
Owner

It will try three times, what causes failure and how often does it happen?

@1223334444abc
Copy link
Author

1223334444abc commented Jul 10, 2023

Once there is any fluctuation in the network that leads to a loss of connection, the program will only retry three times and then exit. (Actually, the network connection will be restored in a very short time)

For errors encountered during image download, the program will retry 999 times according to my settings. However, when retrieving a list, it can only be retried three times.

@elvis972602
Copy link
Owner

I would change it to have the same number of retries and wait intervals as the downloaded file.

@1223334444abc
Copy link
Author

.exe need

@elvis972602
Copy link
Owner

release

@1223334444abc
Copy link
Author

1223334444abc commented Jul 10, 2023

After I specified 'date after: 20230531 update after: 20230531', all posts of this creator will be mistakenly skipped.
It has been tested that whether downloading all favorites or only downloading this creator separately, the post will be skipped.

Because this is the creator of my new focus, I noticed this problem. There may have been other posts mistakenly skipped (although it seems quite normal).

image
(and complaint about the usage instructions.)

@elvis972602
Copy link
Owner

date : publish time
update : edit, update time
date after: 20230531 update after: 20230531 means publish after 20230531 AND edit after 20230531
It will be skepped because the edit date is empty, which means it has not been edited.

@elvis972602
Copy link
Owner

You can use date-after to download a new post and update-after to check if the post has been edited since you downloaded it.
I'll note that in the new document.

@1223334444abc
Copy link
Author

1223334444abc commented Jul 10, 2023

Will I successfully update my database by running these two options separately?

If the logic of "or" can be added, it will greatly improve efficiency. (However, this is not necessary)
Perhaps it can be fixed as an 'incremental backup'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants