Error downloading from MangaDex using latest release

[dykn]

INFO[0000] Downloading... url="https://mangadex.org/title/907/i-am-a-hero"
ERRO[0530] could not get chapter 511880: could not decode response: invalid character '<' looking for beginning of value

Tried multiple different mangas/URLs.

Thanks.

[Girbons]

Hi @dykn, I'll take a look asap!

[Girbons]

@dykn I think that its not related to the tool itself but at some point the magadex server starts to throttle the requests.

@bake what do you think about it?

[dykn]

Is it possible it's an IP address block/ban? The VPS it's on doesn't have an option for a seconday/dynamic IP address, so I don't really have a way of knowing for sure. The error has persisted for over a week now, so I'm just making that assumption at this point.

[Girbons]

I guess so, the error is due to a

 json.Decode(response.Body)

where Body is not a json.

I'll check what is inside Body when this happen but I suppose that there's html inside.

[bake]

Thank you for the mention! I think you're right, @Girbons, I'll add an additional error check tomorrow.

[Girbons]

Thank you, it could be helpful.

[bake]

Please try the latest commit. It only decodes if the status is OK and shows the message otherwise. While testing, I got 503. This reports a HTML page with the message "Please stop spamming the server", however the client replies with the default message of "Service Unavailable".

[Girbons]

Thanks @bake, I'm going to try it out.

[dykn]

Newest error, using latest release:

./comics-downloader -url="https://mangadex.org/title/45112/tensei-kenja-wa-musume-to-kurasu" -format=cbz INFO[0000] Downloading... url="https://mangadex.org/title/45112/tensei-kenja-wa-musume-to-kurasu" ERRO[0000] could not unmarshal manga 45112: json: invalid number literal, trying to unmarshal """" into Number

[bake]

I've just tested it and got the same error. However, using the client directly works:

func main() {
	md := mangadex.New()
	m, _, err := md.Manga("45112")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(m.Title)
	// Tensei Kenja wa Musume to Kurasu
}

[Girbons]

I'll take a look tomorrow on this, also noted that is the same error that I get while running tests on mangadex.

[bake]

FWIW it doesn't work for me with v0.21.2 but it does when I go run (or make) the program directly.

[Girbons]

Same error while running the client directly

[bake]

Might have something to do with MD using Cloudflare again. Could you please try to save the response? With something like httpcache:

c := httpcache.New(diskcache.New("cache", diskcache.NoExpiration))
md := mangadex.New(mangadex.WithHTTPClient(c.Client()))

I also noticed the output before the error message. Are you using the most recent commit?

[Girbons]

The response seems to be ok:

HTTP/2.0 200 OK
Content-Security-Policy: upgrade-insecure-requests;
Content-Type: application/json
Date: Sun, 08 Mar 2020 13:24:21 GMT
Server: ngjit
Set-Cookie: __ddg1=7rR09689CrnSVVdbpEae; Domain=.mangadex.org; HttpOnly; Path=/; Expires=Mon, 08-Mar-2021 13:24:20 GMT
Vary: Accept-Encoding

{"manga":{"cover_url":"\/images\/manga\/45112.jpg?1580472947","description":"Meet Kento Oonari, salaryman in hell.\r\nOverworked by his bosses, his head surely whirled.\r\nHe dies, but meets up with a god called Raidel\r\nWho tells him that he’ll be sent to a new world.\r\n\r\nKento's fairly cautious - he’s read this before.  \r\nAnd so, to the god, he begs for one thing:\r\n“I just want to live a slow life, nothing more.\r\n“Just give me a house and a skill for farming.”\r\n\r\nSo Kento is sent to a medieval age\r\nTo live a nice, slow life. But what a surprise!\r\nThe god gave him the powers of a Great Sage!\r\nAnd Kento used magic in front of commoner eyes!\r\n\r\nSo join us today, as we read of this man:\r\nHis cheat skills, his slow life, and his little girl clan.\r\n\r\n[url=https:\/\/ncode.syosetu.com\/n4302es\/]Original WN[\/url]","title":"Tensei Kenja wa Musume to Kurasu","artist":"Uso Uzaki\u3000","author":"Ryo Kotohira","status":1,"genres":[5,31,41,42,65],"last_chapter":"0","lang_name":"Japanese","lang_flag":"jp","hentai":0,"links":{"ap":"tensei-kenja-wa-musume-to-kurasu","mu":"162804","raw":"https:\/\/seiga.nicovideo.jp\/comic\/45295"}},"chapter":{"816959":{"volume":"","chapter":"2","title":"","lang_code":"fr","group_id":11025,"group_name":"Shiro FanTrad","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582915077},"812849":{"volume":"","chapter":"2","title":"","lang_code":"ru","group_id":11141,"group_name":"S1nGeR","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582389516},"816906":{"volume":"","chapter":"1.3","title":"","lang_code":"fr","group_id":11025,"group_name":"Shiro FanTrad","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582906719},"811874":{"volume":"","chapter":"1.3","title":"","lang_code":"ru","group_id":11141,"group_name":"S1nGeR","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582284641},"815967":{"volume":"","chapter":"1.2","title":"","lang_code":"fr","group_id":11025,"group_name":"Shiro FanTrad","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582755619},"811873":{"volume":"","chapter":"1.2","title":"","lang_code":"ru","group_id":11141,"group_name":"S1nGeR","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582284592},"810596":{"volume":"","chapter":"1.1","title":"","lang_code":"fr","group_id":11025,"group_name":"Shiro FanTrad","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582159890},"811872":{"volume":"","chapter":"1.1","title":"","lang_code":"ru","group_id":11141,"group_name":"S1nGeR","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582284535},"810594":{"volume":"","chapter":"1","title":"","lang_code":"fr","group_id":11025,"group_name":"Shiro FanTrad","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582159819},"811870":{"volume":"","chapter":"1","title":"","lang_code":"ru","group_id":11141,"group_name":"S1nGeR","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582284330},"822479":{"volume":"1","chapter":"2.1","title":"(part two)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1583590563},"812800":{"volume":"1","chapter":"2","title":"(part one)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582383029},"814802":{"volume":"1","chapter":"2","title":"Part 1","lang_code":"id","group_id":6390,"group_name":"Aoirosubs","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582614362},"805995":{"volume":"1","chapter":"1.3","title":"(part four)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1581768408},"814657":{"volume":"1","chapter":"1.3","title":"Part 4","lang_code":"id","group_id":6390,"group_name":"Aoirosubs","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582597982},"805965":{"volume":"1","chapter":"1.2","title":"(part three)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1581767999},"814655":{"volume":"1","chapter":"1.2","title":"Part 3","lang_code":"id","group_id":6390,"group_name":"Aoirosubs","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582597867},"805092":{"volume":"1","chapter":"1.1","title":"(Part two)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1581682832},"809097":{"volume":"1","chapter":"1.1","title":"Part 2","lang_code":"id","group_id":6390,"group_name":"Aoirosubs","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582024322},"803611":{"volume":"1","chapter":"1","title":"(Part one)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1581467928},"808743":{"volume":"1","chapter":"1","title":"Part 1","lang_code":"id","group_id":6390,"group_name":"Aoirosubs","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1581980418}},"status":"OK"}```

[bake]

There seems to have been a breaking change in Go 1.14 which disallows invalid (in this case missing, see volume in chapter) numbers in json.Number. I currently don't have the time to decide and upgrade to int64, strings or a custom type. This also requires breaking changes in the library. There will be an update during the next week and for now you could use 1.13 to build the program.

[bake]

Please try the new commit, I've added a wrapper.

[Girbons]

Going to try the new commit thanks!

[Girbons]

Hey @bake thanks a lot, It works, I'm going to release a new version

[Girbons]

v0.21.3 released