Skip to content

/go

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Get Response body repeated #29955

Closed
ivolkoff opened this issue Jan 27, 2019 · 1 comment

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@ivolkoff @odeke-em
@ivolkoff
Copy link

commented Jan 27, 2019

What version of Go are you using (go version)?

$ go version
go version go1.11.5 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

set GOARCH=amd64

set GOBIN=

set GOCACHE=C:\Users\Ivan\AppData\Local\go-build

set GOEXE=.exe

set GOFLAGS=

set GOHOSTARCH=amd64

set GOHOSTOS=windows

set GOOS=windows

set GOPATH=C:\Users\Ivan\go

set GOPROXY=

set GORACE=

set GOROOT=C:\Go

set GOTMPDIR=

set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64

set GCCGO=gccgo

set CC=gcc

set CXX=g++

set CGO_ENABLED=1

set GOMOD=

set CGO_CFLAGS=-g -O2

set CGO_CPPFLAGS=

set CGO_CXXFLAGS=-g -O2

set CGO_FFLAGS=-g -O2

set CGO_LDFLAGS=-g -O2

set PKG_CONFIG=pkg-config

set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\3C8A~1\AppData\Local\Temp\go-build734602198=/tmp/go-build -gno-record-gcc-switches

What did you do?

https://play.golang.org/p/8oOAL2ZpFYQ

What did you expect to see?

I expect each request to be fulfilled and return a unique page.

What did you see instead?

I see that the body of the page is repeated and I do not find the reason for this.

image

https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006223239&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=27&fastReqId=1

https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006262799&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=28&fastReqId=1

@odeke-em odeke-em changed the title http.get response body repeated net/http: Get Response body repeated Jan 28, 2019

@odeke-em

This comment has been minimized.

Sign in to view
Copy link
Member

commented Jan 28, 2019

Hello there @ivolkoff, thank you filing this bug and welcome to the Go project!

So while visually they look the same, those URLs seem to all have different content.
Just for a quick demonstration of the URLs that you've provided below in

https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006223239&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=27&fastReqId=1

https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006262799&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=28&fastReqId=1

If one runs

curl "https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006223239&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=27&fastReqId=1" \
> a.html \
&& \
curl "https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006262799&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=28&fastReqId=1" \
> b.html \
&& \
diff -u a.html b.html

the result shows entire differences as per https://gist.github.com/odeke-em/ee7d224c23524693630b06587647e2df or inlined

--- a.html	2019-01-28 03:03:17.000000000 -0800
+++ b.html	2019-01-28 03:03:40.000000000 -0800
@@ -14,7 +14,7 @@
 <html lang="en">
 <head>
 	<title>
-		Code de commerce - Article L223-29 | Legifrance</title>
+		Code de commerce - Article R228-49 | Legifrance</title>
 
 	
 	<meta http-equiv="pragma" content="no-cache" >
@@ -117,15 +117,15 @@
 				
 
 
-<a href="/;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3">Accueil</a><span class="path-separator">&gt;</span>
-<a href="/./initRechCodeArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3">Les codes en vigueur</a><span class="path-separator">&gt;</span>
-						<strong>Code de commerce - Article L223-29
+<a href="/;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3">Accueil</a><span class="path-separator">&gt;</span>
+<a href="/./initRechCodeArticle.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3">Les codes en vigueur</a><span class="path-separator">&gt;</span>
+						<strong>Code de commerce - Article R228-49
 				</strong>
 			</p>
 		</div>
 		<div id="content">
 			<h2 class="title">
-				Code de commerce - Article L223-29
+				Code de commerce - Article R228-49
 			</h2>
 
 			<div class="fct_links_top">
@@ -144,7 +144,7 @@
 
 <div class="left" style="clear: both;">
 	
-	<a href="/affichCacheMenu.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?type="><span class="showMenu">Masquer le panneau de navigation</span></a>
+	<a href="/affichCacheMenu.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?type="><span class="showMenu">Masquer le panneau de navigation</span></a>
 </div>
 
 <div class="right">
@@ -162,13 +162,13 @@
 		
 	<!-- Article precedent -->
 	
-		<a href="/affichCodeArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?idArticle=LEGIARTI000006223213&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127&amp;categorieLien=id&amp;oldAction=">&lt;&lt; Article précédent</a>
+		<a href="/affichCodeArticle.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?idArticle=LEGIARTI000006262798&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127&amp;categorieLien=id&amp;oldAction=">&lt;&lt; Article précédent</a>
 	
 	
 	 - 
 	<!-- Article suivant -->
 	
-		<a href="/affichCodeArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?idArticle=LEGIARTI000029946665&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127&amp;categorieLien=id&amp;oldAction=&amp;nbResultRech=">Article suivant &gt;&gt;</a>
+		<a href="/affichCodeArticle.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?idArticle=LEGIARTI000006262800&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127&amp;categorieLien=id&amp;oldAction=&amp;nbResultRech=">Article suivant &gt;&gt;</a>
 	
 	
 	-
@@ -208,21 +208,26 @@
 	
 	
 	<div>
-<a href="/affichCode.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">
+<a href="/affichCode.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">
 <strong>Code de commerce</strong>
 </a>
 </div><ul>
 <li>
-<a href="/affichCode.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?idSectionTA=LEGISCTA000006083281&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">Partie législative</a>
+<a href="/affichCode.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?idSectionTA=LEGISCTA000006083283&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">Partie réglementaire</a>
 <ul>
 <li>
-<a href="/affichCode.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?idSectionTA=LEGISCTA000006113739&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">LIVRE II : Des sociétés commerciales et des groupements d'intérêt économique.</a>
+<a href="/affichCode.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?idSectionTA=LEGISCTA000006113776&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">LIVRE II : Des sociétés commerciales et des groupements d'intérêt économique.</a>
 <ul>
 <li>
-<a href="/affichCode.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?idSectionTA=LEGISCTA000006133176&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">TITRE II : Dispositions particulières aux diverses sociétés commerciales.</a>
+<a href="/affichCode.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?idSectionTA=LEGISCTA000006133252&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">TITRE II : Dispositions particulières aux diverses sociétés commerciales.</a>
 <ul>
 <li>
-<a href="/affichCode.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?idSectionTA=LEGISCTA000006146044&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">Chapitre III : Des sociétés à responsabilité limitée.</a>
+<a href="/affichCode.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?idSectionTA=LEGISCTA000006146232&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">Chapitre VIII : Des valeurs mobilières émises par les sociétés par actions.</a>
+<ul>
+<li>
+<a href="/affichCode.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?idSectionTA=LEGISCTA000006161478&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127">Section 4 : Des titres participatifs.</a>
+</li>
+</ul>
 </li>
 </ul>
 </li>
@@ -242,36 +247,28 @@
 	
 	
 	
-	<div class="titreArt">Article L223-29</div><div class="histoArt"> <ul/>
+	<div class="titreArt">Article R228-49</div><div class="histoArt"> <ul/>
 </div><div class="corpsArt">
-<p/>   Dans les assemblées ou lors des consultations écrites, les décisions sont adoptées par un ou plusieurs associés représentant plus de la moitié des parts sociales.<p/>
-<p/>   Si cette majorité n'est pas obtenue et sauf stipulation contraire des statuts, les associés sont, selon les cas, convoqués ou consultés une seconde fois, et les décisions sont prises à la majorité des votes émis, quel que soit le nombre des votants.<p/>
+<p/>   L'assiette de la partie variable de la rémunération des titres participatifs ne peut être supérieure à 40 % du montant nominal du titre.<p/>
+<p/>   Les éléments retenus pour le calcul de la partie variable de la rémunération sont tirés des comptes annuels approuvés ou, s'il y a lieu, des comptes consolidés.<p/>
 </div><br/><h3>Liens relatifs à cet article</h3><div class="liensArtCita">
 				Cité par:
 				<div class="link_list">
-<a class="liensArtResolu" href="/affichTexteArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?cidTexte=JORFTEXT000031393588&amp;idArticle=JORFARTI000031393598&amp;dateTexte=20151029&amp;categorieLien=cid#JORFARTI000031393598">DÉCRET n°2015-1363 du 27 octobre 2015 - art. 1, v. init.</a>
-<br/>
-<a class="liensArtResolu" href="/affichCodeArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?cidTexte=LEGITEXT000005634379&amp;idArticle=LEGIARTI000031013015&amp;dateTexte=20190128&amp;categorieLien=cid#LEGIARTI000031013015">Code de commerce - art. L223-18 (V)</a>
-<br/>
-<a class="liensArtResolu" href="/affichCodeArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?cidTexte=LEGITEXT000005634379&amp;idArticle=LEGIARTI000006223152&amp;dateTexte=20190128&amp;categorieLien=cid#LEGIARTI000006223152">Code de commerce - art. L223-25 (V)</a>
-<br/>
-<a class="liensArtResolu" href="/affichCodeArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?cidTexte=LEGITEXT000005634379&amp;idArticle=LEGIARTI000006223288&amp;dateTexte=20190128&amp;categorieLien=cid#LEGIARTI000006223288">Code de commerce - art. L223-35 (V)</a>
-<br/>
-<a class="liensArtResolu" href="/affichCodeArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?cidTexte=LEGITEXT000006072050&amp;idArticle=LEGIARTI000031394618&amp;dateTexte=20190128&amp;categorieLien=cid#LEGIARTI000031394618">Code du travail - art. R7331-8 (VD)</a>
+<a class="liensArtResolu" href="/affichCodeArticle.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?cidTexte=LEGITEXT000006072026&amp;idArticle=LEGIARTI000006639629&amp;dateTexte=20190128&amp;categorieLien=cid#LEGIARTI000006639629">Code monétaire et financier - art. D213-26 (V)</a>
 <br/>
 </div>
 <br/>
                 Codifié par:
 				<div class="link_list">
-<a class="liensArtResolu" href="/affichTexte.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?cidTexte=JORFTEXT000000219662&amp;dateTexte=20000921&amp;categorieLien=cid#JORFTEXT000000219662">Ordonnance 2000-912 2000-09-18</a>
-<br/>
-<a class="liensArtResolu" href="/affichTexte.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?cidTexte=JORFTEXT000000595061&amp;dateTexte=20030104&amp;categorieLien=cid#JORFTEXT000000595061">Loi 2003-7 2003-01-03 art. 50 I (ratification)</a>
+<a class="liensArtResolu" href="/affichTexte.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?cidTexte=JORFTEXT000000820567&amp;dateTexte=20070327&amp;categorieLien=cid#JORFTEXT000000820567">Décret 2007-431 2007-03-25 JORF 27 mars 2007</a>
 <br/>
 </div>
 <br/>
                 Anciens textes:
 				<div class="link_list">
-<a class="liensArtResolu" href="/affichTexteArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?cidTexte=JORFTEXT000000692245&amp;idArticle=LEGIARTI000006274700&amp;dateTexte=19660726&amp;categorieLien=id#LEGIARTI000006274700">Loi n°66-537 du 24 juillet 1966 - art. 59 (Ab)</a>
+<a class="liensArtResolu" href="/affichTexteArticle.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?cidTexte=JORFTEXT000000674772&amp;idArticle=LEGIARTI000006344266&amp;dateTexte=19670324&amp;categorieLien=id#LEGIARTI000006344266">Décret n°67-236 du 23 mars 1967 - art. 242-1 (Ab)</a>
+<br/>
+<a class="liensArtResolu" href="/affichTexteArticle.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?cidTexte=JORFTEXT000000674772&amp;idArticle=LEGIARTI000006344266&amp;dateTexte=19670324&amp;categorieLien=id#LEGIARTI000006344266">Décret n°67-236 du 23 mars 1967 - art. 242-1 (Ab)</a>
 <br/>
 </div>
 <br/>
@@ -329,7 +326,7 @@
 
 <div class="left" style="clear: both;">
 	
-	<a href="/affichCacheMenu.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?type="><span class="showMenu">Masquer le panneau de navigation</span></a>
+	<a href="/affichCacheMenu.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?type="><span class="showMenu">Masquer le panneau de navigation</span></a>
 </div>
 
 <div class="right">
@@ -347,13 +344,13 @@
 		
 	<!-- Article precedent -->
 	
-		<a href="/affichCodeArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?idArticle=LEGIARTI000006223213&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127&amp;categorieLien=id&amp;oldAction=">&lt;&lt; Article précédent</a>
+		<a href="/affichCodeArticle.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?idArticle=LEGIARTI000006262798&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127&amp;categorieLien=id&amp;oldAction=">&lt;&lt; Article précédent</a>
 	
 	
 	 - 
 	<!-- Article suivant -->
 	
-		<a href="/affichCodeArticle.do;jsessionid=1901E50917CFEEE07F4AAF9E40AF0882.tplgfr28s_3?idArticle=LEGIARTI000029946665&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127&amp;categorieLien=id&amp;oldAction=&amp;nbResultRech=">Article suivant &gt;&gt;</a>
+		<a href="/affichCodeArticle.do;jsessionid=F4B9A78E34527609AB82A4A954B2F0B4.tplgfr28s_3?idArticle=LEGIARTI000006262800&amp;cidTexte=LEGITEXT000005634379&amp;dateTexte=20190127&amp;categorieLien=id&amp;oldAction=&amp;nbResultRech=">Article suivant &gt;&gt;</a>
 	
 	
 	-
@@ -416,7 +413,7 @@
 <!-- MARQUAGE XiTi  -->
 <script type="text/JavaScript">
 <!--
-xt_form = "&f1=LEGIARTI000006223239&f2=LEGI&f3=Article&f4=L223-29&f5=VIGUEUR&f6=&f7=&f8=&f9=&f10=";
+xt_form = "&f1=LEGIARTI000006262799&f2=LEGI&f3=Article&f4=R228-49&f5=VIGUEUR&f6=&f7=&f8=&f9=&f10=";
 //do not modify below
 if (window.xtparam!=null) {window.xtparam+=xt_form;}
 else {window.xtparam = xt_form;};

To further demo the differences, I've modified your program to create a checksumming utility that will print out the clashing URLs otherwise, "No clashes were detected"

and I've uploaded it to https://play.golang.org/p/1hE9AztcDKB or inlined below and it will print "No clashes were detected"

package main

import (
	"crypto/md5"
	"fmt"
	"io"
	"io/ioutil"
	"net"
	"net/http"
	"strings"
	"sync"
	"time"
)

func main() {
	urls := []string{
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000028743898&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=1&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000032955730&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=2&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000032132092&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=3&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000032944499&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=4&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000022496429&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=5&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000020196876&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=6&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000020196883&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=7&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000026454267&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=8&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000030985098&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=9&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000030985094&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=10&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000030044870&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=11&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000030985126&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=12&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000034182699&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=13&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000006241532&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=14&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000034161980&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=15&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000033461935&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=16&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000033739520&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=17&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000037266553&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=18&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000006219296&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=19&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=91E19D18FF040CFB3D743B58C5D21E12.tplgfr34s_2?idArticle=LEGIARTI000006219127&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=20&fastReqId=1742940652&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006224130&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=21&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006226098&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=22&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006228726&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=23&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006241532&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=24&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006224359&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=25&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006222351&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=26&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006223239&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=27&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006262799&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=28&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006239978&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=29&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006234274&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=30&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006219167&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=31&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006219304&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=32&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006219821&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=33&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006219127&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=34&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006219296&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=35&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006219812&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=36&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006259055&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=37&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006219609&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=38&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006222461&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=39&fastReqId=1338593856&oldAction=rechCodeArticl",
		"https://www.legifrance.gouv.fr/affichCodeArticle.do;jsessionid=3B2A2AFFC85D4227A434B7DACCAD80F0.tplgfr28s_3?idArticle=LEGIARTI000006222473&cidTexte=LEGITEXT000005634379&dateTexte=20190127&fastPos=40&fastReqId=1338593856&oldAction=rechCodeArticl",
	}

	checkDuplicates(processPayloads(urls))
}

func processPayloads(urls []string) (ch <-chan *payload) {
	payloadsCh := make(chan *payload)

	go func() {
		var wg sync.WaitGroup
		defer func() {
			wg.Wait()
			close(payloadsCh)
		}()

		for _, link := range urls {
			wg.Add(1)

			go func(link string) {
				pIn := &payload{
					url: link,
				}
				defer func() {
					payloadsCh <- pIn
					wg.Done()
				}()

				DefaultClient := &http.Client{
					Timeout: time.Duration(5) * time.Second,
					Transport: &http.Transport{
						DisableKeepAlives: true,
						DialContext: (&net.Dialer{
							Timeout:   3 * time.Second,
							KeepAlive: 0,
							DualStack: true,
						}).DialContext,
					},
				}

				res, err := DefaultClient.Get(link)
				if err != nil {
					pIn.err = err
					return
				}
				defer res.Body.Close()

				cksum := md5.New()
				tr := io.TeeReader(res.Body, cksum)
				pIn.body, pIn.err = ioutil.ReadAll(tr)
				pIn.checksum = fmt.Sprintf("%x", cksum.Sum(nil))
			}(link)
		}
	}()

	return payloadsCh
}

type payload struct {
	url      string
	body     []byte
	checksum string
	err      error
}

func checkDuplicates(payloadsIn <-chan *payload) {
	groupingsByChecksum := make(map[string][]string)
	for pIn := range payloadsIn {
		key := pIn.checksum
		value := pIn.url
		groupingsByChecksum[key] = append(groupingsByChecksum[key], value)
	}

	// Finally provide the summary.
	var clashesDetected int
	for checksum, urls := range groupingsByChecksum {
		if len(urls) > 1 {
			fmt.Printf("Clashes detected at checksum %s: with %d URLs as below\n%s\n", checksum,
				strings.Join(append([]string{""}, urls...), "\t\n"))
			clashesDetected++
		}
	}

	if clashesDetected == 0 {
		fmt.Println("No clashes detected!\n")
	}
}

I am going to close this issue as visually-perceptually tricky but actually results are different.

@odeke-em odeke-em closed this Jan 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.