Move github_flavored_markdown to its own repo.

[lithammer]

I briefly mentioned this in #18. My main reason for this is when vendoring (or just go get) it will pull in all of shurcooL/go and all of its dependencies, which is a lot of code (~600 000 lines all in all):

It takes around 1 minute to go get.

$ export GOPATH=$(pwd)
$ time go get github.com/shurcooL/go/github_flavored_markdown
13.50s user 5.71s system 29% cpu 1:05.04 total
$ tree -d -L 3 src
src
├── code.google.com
│   └── p
│       └── rog-go
├── github.com
│   ├── alecthomas
│   │   ├── binary
│   │   └── unsafeslice
│   ├── gogo
│   │   └── protobuf
│   ├── golang
│   │   └── protobuf
│   ├── google
│   │   └── go-querystring
│   ├── gorilla
│   │   └── context
│   ├── kr
│   │   └── fs
│   ├── microcosm-cc
│   │   └── bluemonday
│   ├── russross
│   │   └── blackfriday
│   ├── sergi
│   │   └── go-diff
│   ├── shurcooL
│   │   ├── go
│   │   │   ├── exp
│   │   │   ├── gists
│   │   │   ├── github_flavored_markdown
│   │   │   ├── gopath_util
│   │   │   ├── gopherjs_http
│   │   │   ├── highlight_diff
│   │   │   ├── highlight_go
│   │   │   ├── html_gen
│   │   │   ├── html_table
│   │   │   ├── html_to_markdown
│   │   │   ├── indentwriter
│   │   │   ├── markdown_http
│   │   │   ├── path
│   │   │   ├── pipe_util
│   │   │   ├── raw_file_server
│   │   │   ├── time_util
│   │   │   ├── trash
│   │   │   ├── trim
│   │   │   ├── u
│   │   │   ├── vcs
│   │   │   ├── vfs
│   │   │   └── vfs_util
│   │   └── sanitized_anchor_name
│   ├── smartystreets
│   │   └── mafsa
│   └── sourcegraph
│       ├── annotate
│       ├── go-github
│       ├── mux
│       └── syntaxhighlight
├── golang.org
│   └── x
│       ├── net
│       └── tools
└── sourcegraph.com
    ├── sourcegraph
    │   ├── go-diff
    │   ├── go-nnz
    │   ├── go-sourcegraph
    │   ├── go-vcs
    │   ├── rwvfs
    │   ├── srclib
    │   └── vcsstore
    └── sqs
        └── pbtypes

$ cloc src
    4450 text files.
    4100 unique files.
     958 files ignored.

http://cloc.sourceforge.net v 1.62  T=18.74 s (180.9 files/s, 42963.0 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
Go                                    2298          59247          64469         430953
C                                      485          34337          14465         126243
C/C++ Header                           249           4650          17214          15111
Javascript                              31           1675            847           9644
CSS                                      7           1385             61           6799
HTML                                    91           1316            395           5288
SASS                                    97            528           1159           2830
CMake                                    6            144            155            710
Assembly                                 5             53             26            644
make                                    51            182            985            484
Bourne Shell                            16             73             47            342
JSON                                     5              0              0            329
YAML                                    22             68             24            328
Python                                   6             91             33            261
yacc                                     1             57             52            249
Objective C                              2             34              0            248
Lisp                                     2             34             65            225
Bourne Again Shell                       5             42             26            157
PHP                                      4             12             20            119
vim script                               2             11             27             70
Windows Module Definition                2              0              0             58
Windows Resource File                    1              3              1             40
XML                                      1              0              0             11
Ruby                                     1              4              0              8
---------------------------------------------------------------------------------------
SUM:                                  3390         103946         100071         601151
---------------------------------------------------------------------------------------
cloc src  11.51s user 10.11s system 113% cpu 18.969 total

[dmitshur]

Thanks for opening this issue. I will work on resolving it.

[dmitshur]

The reason it takes one minute to go get from scratch (45 seconds on my machine) and creates such a large tree is a single outlier dependency.

github_flavored_markdown package itself directly imports only a handful of 3rd party packages:

http://godoc.org/github.com/shurcooL/go/github_flavored_markdown?imports

They are:

• github.com/microcosm-cc/bluemonday
  • Package bluemonday provides a way of describing a whitelist of HTML elements and attributes as a policy, and for that policy to be applied to untrusted strings from users that may contain markup.
• github.com/russross/blackfriday
  • Blackfriday markdown processor.
• github.com/shurcooL/go/highlight_diff
  • Package highlight_diff provides syntaxhighlight.Printer and syntaxhighlight.Annotator implementations for diff format.
• github.com/shurcooL/go/highlight_go
  • Package highlight_go provides a syntax highlighter for Go, using go/​scanner.
• github.com/shurcooL/sanitized_anchor_name
  • Package sanitized_anchor_name provides a func to create sanitized anchor names.
• github.com/sourcegraph/annotate
  • Go package for applying multiple sets of annotations to a region of text
• github.com/sourcegraph/syntaxhighlight
  • Package syntaxhighlight provides syntax highlighting for code.

However, if you look at the entire dependency tree,

https://godoc.org/github.com/shurcooL/go/github_flavored_markdown?import-graph&hide=2

It becomes clear that the github.com/sourcegraph/syntaxhighlight package pulls in a massive amount of additional 3rd party packages. Compare syntaxhighlight and annotate, which are similar in code/functionality volume (/cc @gbbr @sqs):

https://godoc.org/github.com/sourcegraph/syntaxhighlight?import-graph&hide=1

vs.

https://godoc.org/github.com/sourcegraph/annotate?import-graph&hide=1

syntaxhighlight used to be much more lightweight, until sourcegraph/syntaxhighlight#11 was merged. That PR added two dependencies sourcegraph.com/sourcegraph/go-sourcegraph/sourcegraph and sourcegraph.com/sourcegraph/vcsstore/vcsclient which are only used for a handful of types that are defined inside, nothing more.

So simply factoring github_flavored_markdown outside into a standalone repo will not fix this problem completely, not until syntaxhighlight is slimmed down as well.

I see two options. Since github_flavored_markdown does not use the new NilAnnotator feature added, it could simply vendor the commit just before that PR. Alternatively, I can look and see if syntaxhighlight can be refactored in a way to avoid pulling in so many dependencies, since it uses only a couple types and does not need the majority of the pulled in functionality/code.

I will think about it more and decide on the best way to improve this situation.

[dmitshur]

Hey @Renstrom,

Now that sourcegraph/syntaxhighlight#15 has been resolved, go getting github_flavored_markdown from scratch takes 8~ seconds instead of 45-60.

[lithammer]

That's great news, thanks for doing this.

[dmitshur]

Ok, I've moved github_flavored_markdown package out of go repo into a dedicated, standalone repo.

See ae0de68.

Old Import Path	New Import Path
github.com/shurcooL/go/github_flavored_markdown	github.com/shurcooL/github_flavored_markdown
github.com/shurcooL/go/highlight_diff	github.com/shurcooL/highlight_diff
github.com/shurcooL/go/highlight_go	github.com/shurcooL/highlight_go

You can and should start using new import path right now. I'll leave the old import path for github_flavored_markdown for a month or so, and then remove it. That should be enough time for people to update.

Time to go get with completely empty GOPATH workspace is a little faster (not a whole lot), but the number of folders is less if you don't need anything else from my go repo.

~ $ export GOPATH=/tmp/tempgopath
~ $ time go get github.com/shurcooL/go/github_flavored_markdown

real    0m19.296s
user    0m1.847s
sys 0m1.029s
~ $ rm -rf /tmp/tempgopath/
~ $ time go get github.com/shurcooL/github_flavored_markdown

real    0m14.762s
user    0m1.841s
sys 0m0.959s
~ $ tree -d -L 3 $GOPATH/src
/tmp/tempgopath/src
├── github.com
│   ├── microcosm-cc
│   │   └── bluemonday
│   ├── russross
│   │   └── blackfriday
│   ├── sergi
│   │   └── go-diff
│   ├── shurcooL
│   │   ├── github_flavored_markdown
│   │   ├── highlight_diff
│   │   ├── highlight_go
│   │   └── sanitized_anchor_name
│   └── sourcegraph
│       ├── annotate
│       └── syntaxhighlight
└── golang.org
    └── x
        └── net

18 directories
~ $ 

Hope that helps!

[dmitshur]

I've created the following pull requests to publicly known packages (via godoc.org/github.com/shurcooL/go/github_flavored_markdown?importers that still import old path to update their import path:

• Update import path of github_flavored_markdown package. Lavos/md2html#1
• Update import path of github_flavored_markdown package. dushmis/go-wiki#1
• Update import path of github_flavored_markdown package. simonjefford/docprinter#1
• https://github.com/ghetzel/sprinkles-api/pull/1
• Update import path of github_flavored_markdown package. hawx/ggg#1
• Update import path of github_flavored_markdown package. hhatto/ftcat#1
• Update import path of github_flavored_markdown package. ipfans/miniwiki#1
• Update envoy-hot-restart-mystery.md jpittis/blog#1
• https://github.com/jpittis/live-markdown/pull/1
• Update import path of github_flavored_markdown package. quintessence/gopress#1
• Update import path of github_flavored_markdown package. yujiod/wiki#4