New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

arXiv #2

Open
davidar opened this Issue Aug 25, 2015 · 26 comments

Comments

Projects
None yet
8 participants
@davidar
Member

davidar commented Aug 25, 2015

TL;DR: Click here.

@davidar davidar added the in progress label Aug 25, 2015

@davidar davidar self-assigned this Aug 25, 2015

@jbenet

This comment has been minimized.

Member

jbenet commented Aug 28, 2015

@rht

This comment has been minimized.

rht commented Sep 15, 2015

Don't have enough network+storage bandwidth to do this, but wish to cat the source archives https://gateway.ipfs.io/ipfs/QmPFiQcfUPr9DYTJ7MPzY9ixxWkfK1oqQQEmQHS1zeD5PP and then process it to get the most used operators 1-gram / var names.

e.g. with 50 sources (from 1 field):

1. { 15425 590.54%
2. _ 9059 346.82%
3. ^ 7225 276.61%
4. - 4081 156.24%
5. + 3019 115.58%
6. = 2612 100.00%
7. \frac 1757 67.27%
8. \mu 1485 56.85%
9. \partial 1295 49.58%
10. \bar 1290 49.39%

If brevity of a notation comes with usage.

@davidar

This comment has been minimized.

Member

davidar commented Sep 15, 2015

@rht what I'd really love to do is http://latexsearch.com

But more generally, running computations on large distributed datasets is an interesting problem. IPMR (map reduce) anyone? It would be cool if you could pay nodes (bitcoin?) to run some sandboxed code over their local blockstore and publish the results back to the ipfs network. @jbenet have you had any thoughts on this topic?

@davidar

This comment has been minimized.

Member

davidar commented Sep 15, 2015

@rht Also, if you could give me the script to do this I'd be happy to run it on pollux

@rht

This comment has been minimized.

rht commented Sep 15, 2015

(on latexsearch : it is too crude. it's like searching through latex "bytecodes" instead of the formulas it represent)

I will put the script here after I have modified it to walk through several files.
Currently it is https://ipfs.io/ipfs/QmQtLp5BxYVgo6miYz3EUAaoP32eqUcW6G8ywsiMxGB5vT

@davidar

This comment has been minimized.

Member

davidar commented Sep 15, 2015

rht: I guess my interest is more academic, as in, has anybody written anything about this expression I've just derived?

Also, would be really cool to finalise this issue soon (and publicise it a little bit). @whyrusleeping what's the best way to proceed with adding lots of small files at the moment? Edit: use the tar adder? Are you able to do something like ipfs cat /ipfs/hash/archive.tar/foo/bar.txt?

Edit: and also do cool stuff with it to showcase the benefits of publishing with a creative commons license. CC-BY-SA is really the embodiment of the scientific social contract

@rht

This comment has been minimized.

rht commented Sep 15, 2015

But it would be better if there are specifiers e.g. if the expression is vectorized, approximate/exact, etc.

https://ipfs.io/ipfs/QmcH6av29y7fXBqifmEMU54vVuMYjpBFBdcQsDkWZK8Hbi should work with python parselatex.py dirname.
Also wanted to know how long it takes to traverse the entire CC arxiv.

@davidar

This comment has been minimized.

Member

davidar commented Sep 15, 2015

@rht definitely, and that's exactly why an open source version of LaTeX search would be better ☺

@davidar

This comment has been minimized.

Member

davidar commented Sep 15, 2015

@jbenet

This comment has been minimized.

Member

jbenet commented Sep 15, 2015

We should get @rht access on pollux -- @rht what's your ssh pubkey? 


Sent from Mailbox

On Tue, Sep 15, 2015 at 6:13 AM, David A Roberts notifications@github.com
wrote:

@rht https://ipfs.io/ipfs/QmeFkmYSPhn33hzEHxxqtvDNxHorW93iZFFMgZDQKrfeH4

real 6m24.104s
user 3m34.448s

sys 1m14.448s

Reply to this email directly or view it on GitHub:
#2 (comment)

@rht

This comment has been minimized.

rht commented Sep 15, 2015

That's too fast!

This would have taken ages if I were to do it by myself (or if anyone had done this before).

It appears that

  • \sum / \int (map-reduce of +) is the most used operator (\partial doesn't count because it appears 3 or 4 times in a line). This could have been matmul, but unfortunately this is implicit in latex.
  • \alpha is actually being used more than \mu.
  • \rangle happens more often than \psi (by a small margin)
@rht

This comment has been minimized.

rht commented Sep 15, 2015

@jbenet https://api.github.com/users/rht/keys, the one with id 12491781, as @lgierth suggested.

@whyrusleeping

This comment has been minimized.

Member

whyrusleeping commented Sep 15, 2015

@davidar the IPMR thing sounds like it could be accomplished with ethereum and ipfs.

@whyrusleeping

This comment has been minimized.

Member

whyrusleeping commented Sep 15, 2015

@whyrusleeping what's the best way to proceed with adding lots of small files at the moment? Edit: use the tar adder? Are you able to do something like ipfs cat /ipfs/hash/archive.tar/foo/bar.txt?

The most reliable way is going to be to add the content offline. Even the tar add command will hit the file descriptor issue :/

@lgierth

This comment has been minimized.

Member

lgierth commented Sep 15, 2015

We should get @rht access on pollux

Will take care of it this sprint: ipfs/infrastructure#87

@jbenet

This comment has been minimized.

Member

jbenet commented Sep 15, 2015

it would be cool if you could pay nodes (bitcoin?) to run some sandboxed code over their local blockstore and publish the results back to the ipfs network. @jbenet have you had any thoughts on this topic?

that's like phase 3 of the IPPlan. (we're in phase 1). we could run some experiments now with this. it shouldn't be too hard.

@davidar

This comment has been minimized.

Member

davidar commented Sep 16, 2015

The most reliable way is going to be to add the content offline. Even the tar add command will hit the file descriptor issue :/

@whyrusleeping I am, still too slow :(

Re ethereum, can you elaborate? I've played with it a little bit, and it seems incapable of doing any kind of interesting computation before heat death of the universe...

@jbenet cool! What's phase 2?

@davidar

This comment has been minimized.

Member

davidar commented Sep 20, 2015

There's still a little bit of work that needs to be done, but I'm going to go ahead and call this the first complete (pdf+src+metadata) release of the arXiv archive:

https://ipfs.io/ipfs/QmfXH9XtP7xmoTH8WAp4HNSduqWMwLTH8B8TvbTkdgzNAa

🎈 🎆

@jbenet

This comment has been minimized.

Member

jbenet commented Sep 23, 2015

🌟 🌠 🎈 👍

@davidar davidar referenced this issue Sep 30, 2015

Open

arXiv #1

@davidar davidar referenced this issue Dec 15, 2015

Open

pub/sub - publish / subscribe #64

0 of 3 tasks complete
@NDuma

This comment has been minimized.

NDuma commented Jan 25, 2016

Link to GitHub via GitXiv.com

@davidar

This comment has been minimized.

Member

davidar commented Jan 26, 2016

@NDuma GitXiv looks really cool, thanks! cc @samim23 @mekarpeles

@rht

This comment has been minimized.

rht commented Jan 16, 2017

@davidar Request for reproducible build script. The spec for the package metadata (which includes the build process) should be achievable at 25 GB-scale (~ one server-node) regardless of the complication for many server-nodes case.

@davidar

This comment has been minimized.

Member

davidar commented Jan 16, 2017

@rht I'm not sure there is a script (yet), it was mostly a (semi-)manual process... :/

@rht

This comment has been minimized.

rht commented Jan 16, 2017

I see. I have re-created the scholarpedia.org archive in a local branch, since this is ~1GB, fits within one node of a computer. I wasn't able to find the example _Metadata.json from @eminence since it looks like the content has been gc-ed.

@leerspace

This comment has been minimized.

Contributor

leerspace commented Mar 11, 2017

The hash for this on archives.ipfs.io (/ipfs/QmfXH9XtP7xmoTH8WAp4HNSduqWMwLTH8B8TvbTkdgzNAa) seems to have become unavailable. If anyone still has a copy I can pin it to my node which I plan to keep running indefinitely.

@whyrusleeping

This comment has been minimized.

Member

whyrusleeping commented Mar 11, 2017

I may have a copy on my NAS box which is currently in a storage unit (Recently moved). If nobody comes forth with another copy in the next couple months, i'll venture out to get it and plug it into a fat pipe

@NDuma NDuma referenced this issue Apr 13, 2017

Open

GitXiv #148

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment