Adding new graphs from SNAP

[Azzaare]

I am working with a couple of graphs (citations ones) from SNAP at the moment, so I should take some time to add them here. Is that OK?

I see that the file extension is .jlg (Julia LightGraphs ?). Are they generated with the savegraph function (with the compress=true argument)?

Let me know so that I can make a PR.

[sbromberger]

1. That would be great. Please do add them.
2. Make sure they're compressed (it's on by default). I picked jlg because that's what I started using, but the docs suggest a different extension. It doesn't really matter as long as they can be loaded via loadgraph, but let's keep it consistent.

[Azzaare]

Those networks are slightly tricky since there are some nodes meta-data available on SNAP and the time to convert them (particularly the patents one) to LG format is quite long. So I took the liberty to do the following things.

1. Add .csv files that keeps correspondance between the nodes ids in LG format and the articles/patents original ids. They can be found as x_ids.csv for any file x.jlg. People can match the meta-data if they need it. Maybe an export to other graph format that keeps the meta-data could be done in the future.
2. Put below the code I used to generate those graphs and the associated ids, just in case.

As soon as the conversion to LG compressed format is done (and tested), I will make the PR.

using LightGraphs
using DataFrames

function parsegraph(filepath::AbstractString, nodes::Int)
    f = open(filepath)
    lines = readlines(f)

    ids = Dict{Int, Int}()
    current = 0

    g = DiGraph(nodes)

    for line in lines[5:end]
        st = split(line, '\t')
        s, t = parse(st[1]), parse(st[2])
        if !haskey(ids, s)
            current += 1
        end
        source = get!(ids, s, current)
        if !haskey(ids, t)
            current +=1
        end
        target = get!(ids, t, current)
        add_edge!(g, source, target)
    end

    close(f)

    return g, ids
end

hep_th, ids_th = parsegraph("/data/datasets/snap/cit-HepTh.txt", 27770)
hep_ph, ids_ph = parsegraph("/data/datasets/snap/cit-HepPh.txt", 34546)
patent, ids_pa = parsegraph("/data/datasets/snap/cit-Patents.txt", 3774768)

path = "/your/path/to/SNAPDatasets.jl/"
savegraph(path * "arxiv_cit/arxiv_cit_hep_th.jlg", hep_th, compress=true)
savegraph(path * "arxiv_cit/arxiv_cit_hep_ph.jlg", hep_ph, compress=true)
savegraph(path * "patent_cit/patent_cit_us.jlg", patent, compress=true)

df_th = DataFrame(
    NODE_ID     = [n for n in keys(ids_th)],
    ARTICLE_ID  = [a for a in values(ids_th)])
df_ph = DataFrame(
    NODE_ID     = [n for n in keys(ids_ph)],
    ARTICLE_ID  = [a for a in values(ids_ph)])
df_patents = DataFrame(
    NODE_ID     = [n for n in keys(ids_pa)],
    PATENT_ID   = [p for p in values(ids_pa)])

writetable(path * "arxiv_cit/arxiv_cit_hep_th_ids.csv", df_th)
writetable(path * "arxiv_cit/arxiv_cit_hep_ph_ids.csv", df_ph)
writetable(path * "patent_cit/patent_cit_us_ids.csv", df_patents)

[Azzaare]

Also, I don't think those are necessary to keep in the tests, but also, just in case ...

using SNAPDatasets
using Base.Test

@testset "SNAP Datasets" begin
  @test @inferred "$(loadsnap(:as_caida))" == "{26475, 53381} undirected simple Int64 graph"
  @test @inferred "$(loadsnap(:arxiv_cit_hep_ph))" == "{34546, 421578} directed simple Int64 graph"
  @test @inferred "$(loadsnap(:arxiv_cit_hep_th))" == "{27770, 352807} directed simple Int64 graph"
  @test @inferred "$(loadsnap(:patent_cit_us))" == "{3774768, 16518948} directed simple Int64 graph"
  @test_throws ErrorException loadsnap()
  @test_throws ErrorException loadsnap(:badsnapxxxxx)
end

[sbromberger]

Very nice. Thank you, @Azzaare !