Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Version 20230807.12. Do not connect with the IPs that are unresponsive.
  • Loading branch information
@Arkiver2
Arkiver2 committed Aug 7, 2023
1 parent fa93a1d commit 02966fc
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion pipeline.py
View file
Expand Up @@ -59,7 +59,7 @@
#
# Update this each time you make a non-cosmetic change.
# It will be added to the WARC files and reported to the tracker.
VERSION = '20230807.11'
VERSION = '20230807.12'
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'
TRACKER_ID = 'skyblog'
TRACKER_HOST = 'legacy-api.arpa.li'
Expand Down Expand Up @@ -240,6 +240,8 @@ def realize(self, item):
'--hosts-file', '/dev/null',
'--resolvconf-file', '/dev/null',
'--dns-servers', '9.9.9.10,149.112.112.10,2620:fe::10,2620:fe::fe:10',
'--reject-reserved-subnets',
'--reject-subnets', '91.203.187.104,91.203.187.14,91.203.187.40,91.203.187.70,91.203.187.32,91.203.187.6',
'--load-cookies', 'cookies.txt',
'--content-on-error',
'--no-http-keep-alive',
Expand Down

2 comments on commit 02966fc

@Jalle19
Copy link

@Jalle19 Jalle19 commented on 02966fc Aug 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

91.203.187.96 needs to be added to the list

@Jalle19
Copy link

@Jalle19 Jalle19 commented on 02966fc Aug 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the servers respond just fine, but they ban your source address if you make too many requests. So I don't think this is a sustainable approach since these IPs may work for others.

Please sign in to comment.