Difference between CDN and clojars.org/repo

[martinklepsch]

See:

http://repo.clojars.org/ivarref/datomic-schema/
https://clojars.org/repo/ivarref/datomic-schema/

Specifically versions 0.1.4 and 0.1.4-SNAPSHOT appear to be missing from the CDN distribution.

[danielcompton]

Hmm, this is definitely a bug, the two repositories should show the same files (though they are pointing to two different locations, so getting out of sync like this is possible).

[tobias]

Agreed. And it is an odd one - we write to the cloudfiles repo here: https://github.com/clojars/clojars-web/tree/master/src/clojars/routes/repo.clj#L271, then write the jar to the db here: https://github.com/clojars/clojars-web/tree/master/src/clojars/routes/repo.clj#L278, so if the cloudfile write fails, it should never reach the db write. But 0.1.4 and 0.1.4-SNAPSHOT exist in the db, since they show on https://clojars.org/ivarref/datomic-schema/versions/0.1.4

[tobias]

I scanned all of the jars + poms on disk and checked them against the CDN. The following 17 were missing from cloudfiles (though trying to load the auth artifacts results in a 401 instead of a 404, so may be caused by a configuration issue either in fast.ly or cloudfiles where the auth path has special meaning):

/mnt/repovolume/clojars-repo/auth/auth/0.1.0/auth-0.1.0.jar
/mnt/repovolume/clojars-repo/auth/auth/0.1.0/auth-0.1.0.pom
/mnt/repovolume/clojars-repo/marshallbrekka/clj-aws-s3/0.3.5a/clj-aws-s3-0.3.5a.jar
/mnt/repovolume/clojars-repo/io/github/theasp/simple-encryption/0.1.0/simple-encryption-0.1.0.jar
/mnt/repovolume/clojars-repo/ivarref/datomic-schema/0.1.4-SNAPSHOT/datomic-schema-0.1.4-20181018.144401-1.pom
/mnt/repovolume/clojars-repo/ivarref/datomic-schema/0.1.4-SNAPSHOT/datomic-schema-0.1.4-20181018.144401-1.jar
/mnt/repovolume/clojars-repo/ivarref/datomic-schema/0.1.4/datomic-schema-0.1.4.jar
/mnt/repovolume/clojars-repo/ivarref/datomic-schema/0.1.4/datomic-schema-0.1.4.pom
/mnt/repovolume/clojars-repo/thinktopic/cortex-caffe/0.9.17/cortex-caffe-0.9.17.pom
/mnt/repovolume/clojars-repo/thinktopic/cortex-caffe/0.9.2/cortex-caffe-0.9.2.jar
/mnt/repovolume/clojars-repo/bridg/sqs/0.2.4/sqs-0.2.4.pom
/mnt/repovolume/clojars-repo/bridg/sqs/0.2.0-SNAPSHOT/sqs-0.2.0-20171214.205610-2.jar
/mnt/repovolume/clojars-repo/org/msync/lucene-clj/0.1.0-SNAPSHOT/lucene-clj-0.1.0-20181018.144421-21.pom
/mnt/repovolume/clojars-repo/org/msync/lucene-clj/0.1.0-SNAPSHOT/lucene-clj-0.1.0-20181018.144421-21.jar
/mnt/repovolume/clojars-repo/com/danboykis/affable-async/0.1.0-SNAPSHOT/affable-async-0.1.0-20181018.144728-5.pom
/mnt/repovolume/clojars-repo/com/danboykis/affable-async/0.1.0-SNAPSHOT/affable-async-0.1.0-20181018.144728-5.jar
/mnt/repovolume/clojars-repo/fnparse/fnparse/3.?.4/fnparse-3.?.4.jar
/mnt/repovolume/clojars-repo/fnparse/fnparse/3.?.4/fnparse-3.?.4.pom

[glts]

I have discovered another difference that could be blocking technomancy/leiningen#2581.

Maven metadata files served at clojars.org/repo and repo.clojars.org differ. Example:

• https://clojars.org/repo/org/clojars/glts/deploytest/maven-metadata.xml: ok

• https://repo.clojars.org/org/clojars/glts/deploytest/maven-metadata.xml: 0.4.0-SNAPSHOT is missing!

• https://clojars.org/repo/org/clojars/glts/deploytest/0.4.0-SNAPSHOT/maven-metadata.xml: ok

• https://repo.clojars.org/org/clojars/glts/deploytest/0.4.0-SNAPSHOT/maven-metadata.xml: not served, 404 Not Found!

[glts]

Ah, now they are there. Apparently it takes a few minutes until the changes appear in repo.clojars.org. Apologies for the noise!

[tobias]

@glts correct - there is a 15 minute TTL on the CDN, so changed files may take up to that time to appear in repo.clojars.org.

[tobias]

I believe this is caused by failures in our code that uploads to cloudfiles asynchronously. I'm working on changing the deploy logic to upload synchronously instead, so if uploading fails, the deployment will fail. That should prevent this from happening in the future. I also uploaded any files that were missing from or had an incorrect checksum in the cloudfiles repo (treating the on-disk repo as the canonical source) last week. I'll run that again once I release the sync upload logic to catch any that may have been missed in the interim.

[tobias]

I've rolled out the fix that should prevent this in the future (as part of Clojars 77), so I'm closing this.