Documentation improvements

README.md

@@ -8,82 +8,125 @@
 FilePathsBase.jl provides a type based approach to working with filesystem paths in julia.
 
 ## Intallation
+
 FilePathsBase.jl is registered, so you can to use `Pkg.add` to install it.
-```julia-repl
+```julia
 julia> Pkg.add("FilePathsBase")
 ```
 
-## Usage
-```julia-repl
-julia> using FilePathsBase; using FilePathsBase: /
-```
+## Getting Started
 
-The first important difference about working with paths in FilePathsBase.jl is that path
-segments are represented as an immutable tuple of strings.
+Here are some common operations that you may want to perform with file paths.
 
-Path creation:
-```julia-repl
-julia> Path("~/repos/FilePathsBase.jl/")
-p"~/repos/FilePathsBase.jl/"
-```
-or
-```julia-repl
-julia> p"~/repos/FilePathsBase.jl/"
-p"~/repos/FilePathsBase.jl/"
-```
+```julia
+#=
+NOTE: We're loading our `/` operator for path concatenation into the currect scope, but non-path division operations will still fallback to the base behaviour.
+=#
+julia> using FilePathsBase; using FilePathsBase: /
 
-Human readable file status info:
-```julia-repl
-julia> stat(p"README.md")
+julia> cwd()
+p"/Users/rory/repos/FilePathsBase.jl"
+
+julia> walkpath(cwd() / "docs") |> collect
+23-element Array{Any,1}:
+ p"/Users/rory/repos/FilePathsBase.jl/docs/.DS_Store"
+ p"/Users/rory/repos/FilePathsBase.jl/docs/Manifest.toml"
+ p"/Users/rory/repos/FilePathsBase.jl/docs/Project.toml"
+ p"/Users/rory/repos/FilePathsBase.jl/docs/build"
+ p"/Users/rory/repos/FilePathsBase.jl/docs/build/api.html"
+ p"/Users/rory/repos/FilePathsBase.jl/docs/build/assets"
+ p"/Users/rory/repos/FilePathsBase.jl/docs/build/assets/arrow.svg"
+ p"/Users/rory/repos/FilePathsBase.jl/docs/build/assets/documenter.css"
+ ...
+
+julia> stat(p"docs/src/index.md")
 Status(
-  device = 16777220,
-  inode = 48428965,
+  device = 16777223,
+  inode = 32240108,
   mode = -rw-r--r--,
   nlink = 1,
-  uid = 501,
-  gid = 20,
+  uid = 501 (rory),
+  gid = 20 (staff),
   rdev = 0,
-  size = 1880 (1.8K),
+  size = 2028 (2.0K),
   blksize = 4096 (4.0K),
   blocks = 8,
-  mtime = 2016-02-16T00:49:27,
-  ctime = 2016-02-16T00:49:27,
+  mtime = 2020-04-20T17:20:38.612,
+  ctime = 2020-04-20T17:20:38.612,
 )
-```
 
-Working with permissions:
-```julia-repl
-julia> m = mode(p"README.md")
--rw-r--r--
+julia> relative(p"docs/src/index.md", p"src/")
+p"../docs/src/index.md"
 
-julia> m - readable(:ALL)
---w-------
+julia> normalize(p"src/../docs/src/index.md")
+p"docs/src/index.md"
 
-julia> m + executable(:ALL)
--rwxr-xr-x
+julia> absolute(p"docs/src/index.md")
+p"/Users/rory/repos/FilePathsBase.jl/docs/src/index.md"
 
-julia> chmod(p"README.md", "+x")
+julia> islink(p"docs/src/index.md")
+true
 
-julia> mode(p"README.md")
--rwxr-xr-x
+julia> canonicalize(p"docs/src/index.md")
+p"/Users/rory/repos/FilePathsBase.jl/README.md"
 
-julia> chmod(p"README.md", m)
+julia> parents(p"./docs/src")
+2-element Array{PosixPath,1}:
+ p"."
+ p"./docs"
 
-julia> m = mode(p"README.md")
--rw-r--r--
+julia> parents(absolute(p"./docs/src"))
+6-element Array{PosixPath,1}:
+ p"/"
+ p"/Users"
+ p"/Users/rory"
+ p"/Users/rory/repos"
+ p"/Users/rory/repos/FilePathsBase.jl"
+ p"/Users/rory/repos/FilePathsBase.jl/docs"
 
-julia> chmod(p"README.md", user=(READ+WRITE+EXEC), group=(READ+WRITE), other=READ)
+julia> absolute(p"./docs/src")[1:end-1]
+("Users", "rory", "repos", "FilePathsBase.jl", "docs")
 
-julia> mode(p"README.md")
--rwxrw-r--
+julia> tmpfp = mktempdir(SystemPath)
+p"/var/folders/vz/zx_0gsp9291dhv049t_nx37r0000gn/T/jl_1GCBFT"
 
-```
+julia> sync(p"/Users/rory/repos/FilePathsBase.jl/docs", tmpfp / "docs")
+p"/var/folders/vz/zx_0gsp9291dhv049t_nx37r0000gn/T/jl_1GCBFT/docs"
+
+julia> exists(tmpfp / "docs" / "make.jl")
+true
+
+julia> m = mode(tmpfp / "docs" / "make.jl")
+Mode("-rw-r--r--")
+
+julia> m - readable(:ALL)
+Mode("--w-------")
+
+julia> m + executable(:ALL)
+Mode("-rwxr-xr-x")
+
+julia> chmod(tmpfp / "docs" / "make.jl", "+x")
+"/var/folders/vz/zx_0gsp9291dhv049t_nx37r0000gn/T/jl_1GCBFT/docs/make.jl"
+
+julia> mode(tmpfp / "docs" / "make.jl")
+Mode("-rwxr-xr-x")
+
+# Count LOC
+julia> mapreduce(+, walkpath(cwd() / "src")) do x
+           extension(x) == "jl" ? count("\n", read(x, String)) : 0
+       end
+3020
+
+# Concatenate multiple files.
+julia> str = mapreduce(*, walkpath(tmpfp / "docs" / "src")) do x
+           read(x, String)
+       end
+"# API\n\nAll the standard methods for working with paths in base julia exist in the FilePathsBase.jl. The following describes the rough mapping of method names. Use `?` at the REPL to get the documentation and arguments as they may be different than the base implementations.\n\n..."
+
+# Could also write the result to a file with `write(newfile, str)`)
 
-Reading and writing directly to file paths:
-```julia-repl
-julia> write(p"testfile", "foobar")
-6
+julia> rm(tmpfp; recursive=true)
 
-julia> read(p"testfile")
-"foobar"
+julia> exists(tmpfp)
+false
 ```

docs/make.jl

@@ -1,12 +1,20 @@
 using Documenter, FilePathsBase, FilePathsBase.TestPaths
 
+```@meta
+DocTestSetup = quote
+    using FilePathsBase; using FilePathsBase: /, join;
+end
+```
+
 makedocs(
     modules=[FilePathsBase],
     format=Documenter.HTML(
         prettyurls = get(ENV, "CI", nothing) == "true",
     ),
     pages=[
         "Home" => "index.md",
+        "Design" => "design.md",
+        "FAQ" => "faq.md",
         "API" => "api.md",
     ],
     repo="https://github.com/rofinn/FilePathsBase.jl/blob/{commit}{path}#L{line}",

docs/src/design.md

@@ -0,0 +1,44 @@
+# [Design](@id design_header)
+
+FilePaths.jl and FilePathsBase.jl have gone through several design iterations over the years.
+To help get potential contributors up-to-speed, we'll cover several background points and design choices.
+Whenever possible, we'll reference existing resources (e.g., GitHub issues, blog posts, documentation, software packages) for further reading.
+
+## Filesystem Abstractions
+
+While filesystems themselves are abstractions for data storage, many programming languages
+provide APIs for writing generic/cross-platform software.
+Typically, these abstractions can be broken down into string or typed based solutions.
+
+### String APIs:
+
+- Python: [`os.path`](https://docs.python.org/3.8/library/os.path.html)
+- Haskell: [`System.FilePath`](https://hackage.haskell.org/package/filepath-1.4.2.1/docs/System-FilePath.html)
+- Julia: [`Base.Filesystem`](https://docs.julialang.org/en/v1/base/file/)
+
+This approach tends to be simpler and only requires adding utility methods for interacting with filesystems. Unfortunately, any operations require significant string manipulation to work, and it often cannot be extended for remote filesystems (e.g., S3, FTP, HTTP). Enforcing path validity becomes difficult when any string operation can be applied to the path type (e.g., `join(prefix, segments...)` vs `joinpath(prefix, segments...)`).
+
+### Typed APIs:
+
+- Python: [`pathlib`](https://docs.python.org/3/library/pathlib.html)
+- Rust: [`std::path`](https://doc.rust-lang.org/std/path/index.html)
+- C++: [`std::filesystem`](https://en.cppreference.com/w/cpp/filesystem/path)
+- Haskell: [`path`](https://hackage.haskell.org/package/path)
+- Scala: [`os-lib`](https://github.com/lihaoyi/os-lib)
+
+The primary idea is that a filesystem path is just a sequence of path segments, and so very few path operations overlap with string operations.
+For example, you're unlikely to call string functions like `join(...)`, `chomp(...)`, `eachline(...)`, `match(regex, ...)` or `parse(Float64, ...)` with a filesystem path.
+Further, differentiating strings and paths allows us to define different equality rules and dispatch behaviour on filepaths in our APIs.
+Finally, by defining a common API for all `AbstractPaths`, we can write generic functions that work with `PosixPath`s, `WindowsPath`s, `S3Path`s, `FTPPath`s, etc.
+
+## Path Types
+
+In FilePathsBase.jl, file paths are first and foremost a type that wraps a tuple of strings, representing path `segments`.
+Most path types will also include a `root`, `drive` and `separator`.
+Concrete path types should either directly subtype `AbstractPath` or in the case of local filesystems (e.g., `PosixPath`, `WindowsPath`) from `SystemPath`, as shown in the diagram below.
+
+![Hierarchy](hierarchy.svg)
+
+Notice that our `AbstractPath` type no longer subtypes `AbstractString` like some other libraries.
+We [chose](https://github.com/rofinn/FilePathsBase.jl/issues/15) drop string subtyping because not all `AbstractString` operations make sense on paths, and even more seem like they should perform a fundamentally different operation as mentioned above.
+Similar points have been made for [why `pathlib.Path` doesn't inherit from `str` in Python](https://snarky.ca/why-pathlib-path-doesn-t-inherit-from-str/).

docs/src/faq.md

@@ -0,0 +1,62 @@
+# FAQ
+
+Here we have a growing list of common technical and design questions folks have raised in the past.
+If you feel like something is missing, please open an issue or pull request to add it.
+
+**Q. Should I depend on FilePathsBase.jl and FilePaths.jl?**
+
+A. FilePathsBase.jl is a lightweight dependency for packages who need to operate on `AbstractPath` types and don't need to interact with other packages (e.g., Glob.jl, URIParser.jl, FileIO.jl).
+FilePaths.jl extends FilePathsBase.jl to improve package interop across the Julia ecosystem, at the cost of extra dependencies.
+In general, FilePathsBase.jl should be used for general (low level) packages. While scripts and application-level packages should use FilePaths.jl.
+
+**Q. What's wrong with strings?**
+
+A. In many cases, nothing.
+For local filesystem paths, there's often no functional difference between using an `AbstractPath` and a `String`.
+Some cases where the path type distinction is useful include:
+
+- Path specific operations (e.g., `join`, `/`, `==`)
+- Dispatch on paths vs strings (e.g., `project(name::String) = project(DEFAULT_ROOT / name)`)
+
+See [design section](@ref design_header) for more details on the advantages of path types over strings.
+
+**Q. Why is `AbstractPath` not a subtype of `AbstractString`?
+
+A. Initially, we made `AbstractPath` a subtype of `AbstractString`, but falling back to string operations often didn't make sense (e.g., `ascii(::AbstractPath)`, `chomp(::AbstractPath)`, `match(::Regex, ::AbstractPath)`, `parse(::Type{Float64}, ::AbstractPath)`).
+Having a distinct path type results in fewer confusing error messages and more explicit code (via type conversions). [Link to issue/PR and blog post]
+
+**Q. Why don't you concatenate paths with `*`?**
+
+A. By using `/` for path concatenation (`joinpath`), we can continue to support string concatenation with `*`:
+
+```julia
+julia> cwd() / "src" / "FilePathsBase" * ".jl"
+p"/Users/rory/repos/FilePathsBase.jl/src/FilePathsBase.jl
+```
+
+**Q. How do I write code that works with strings and paths?**
+
+A. FilePathsBase.jl intentionally provides aliases for `Base.Filesystem` functions, so you can perform base filesystem operations on strings and paths interchangeable.
+If something is missing please open an issue or pull request.
+Here are some more concrete tips to help you write generic code:
+- Don't [overly constrain](https://white.ucc.asn.au/2020/04/19/Julia-Antipatterns.html#over-constraining-argument-types) your argument types.
+- Avoid manual string manipulations (e.g., `match`, `replace`).
+- Stick to the overlapping base filesystem aliases (e.g., `joinpath` vs `/`, `normpath` vs `normalize`).
+
+NOTE: The first 2 points are just general best practices independent of path types.
+Unfortunately, the last point is a result of the `Base.Filesystem` API (could change if FilePathsBase.jl becomes a stdlib).
+
+See the usage guide for examples.
+
+**Q: FilePathsBase doesn't work with package X?**
+
+A: In many cases, filepath types and strings are interchangable, but if a specific package constrains the argument type (e.g., `AbstractString`, `String`) then you'll get a `MethodError`.
+There are a few solutions to this problem.
+
+1. Loosen the argument type constraint in the given package.
+2. Add a separate dispatch for `AbstractPath` and add a dependency on FilePathsBase.jl.
+3. For very general/lightweight packages we can add the dependency to FilePaths.jl and extend the offending function there.
+4. Manually convert your path to a string before calling into the package.
+You may need to parse any returned paths to back to a filepath type if necessary.
+
+NOTE: For larger packages, FilePaths.jl provides an `@convert` macro which will handle generating appropriate conversion methods for you.