Make this just load all the different pieces of Str & Chr support

Author: ScottPJones

.travis.yml

@@ -27,9 +27,11 @@ git:
 
 ## uncomment the following lines to override the default test script
 script:
-  - julia -e 'Pkg.clone(pwd()); Pkg.test("Strs"; coverage=true)'
+  - if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
+  - julia -e 'VERSION < v"0.7.0-DEV" || (using Pkg); p="https://github.com/JuliaString"; s=".jl.git"; l="_Entities"; for n in ("StrTables", "LightXML", "JSON", "Format", "PCRE2", "InternedStrings") ; Pkg.add(n) ; end ; t=("LaTeX","Emoji","HTML","Unicode"); for n in t; Pkg.add("$n$l"); end; for n in ("APITools", "StrAPI", "CharSetEncodings", "Chars", "StrBase", "StrRegex", "StrLiterals", "StrFormat", "StrEntities") ; Pkg.clone("$p/$n$s"); end ; Pkg.clone(pwd()); for n in t; Pkg.build("$n$l"); end; Pkg.test("Strs"; coverage=true)'
+  #- julia -e 'Pkg.clone(pwd()); Pkg.test("Strs"; coverage=true)'
 after_success:
   # push coverage results to Coveralls
-  - julia -e 'cd(Pkg.dir("Strs")); Pkg.add("Coverage"); using Coverage; Coveralls.submit(Coveralls.process_folder())'
+  - julia -e 'VERSION < v"0.7.0-DEV" || (using Pkg); cd(Pkg.dir("Strs")); Pkg.add("Coverage"); using Coverage; Coveralls.submit(Coveralls.process_folder())'
   # push coverage results to Codecov
-  - julia -e 'cd(Pkg.dir("Strs")); Pkg.add("Coverage"); using Coverage; Codecov.submit(Codecov.process_folder())'
+  - julia -e 'VERSION < v"0.7.0-DEV" || (using Pkg); cd(Pkg.dir("Strs")); Pkg.add("Coverage"); using Coverage; Codecov.submit(Codecov.process_folder())'

IDEAS.md

@@ -0,0 +1,60 @@
+Ideas:
+Have substring support built in
+
+Have types using both a UniStr type along with UTF-8 and/or UTF-16 versions, and take advantage of
+knowing the real length of the string from the UniStr version, and always reading the string,
+except for times when a UTF-8 (or UTF-16) version is needed, using the O(1) version.
+
+Have extra parameter(s?), for types to hold things like: UnitRange{UInt16,UInt32,UInt64},
+for substrings, UInt64 for caching hash value, Str{T}, where T could be Raw*, UTF8, or UTF16,
+to cache either unchanged input data and/or validated UTF8/UTF16, and possibly even one of those with a substring type!
+
+Add types _UCS2 and _UTF32, for the internal versions of those encodings, which are guaranteed to
+have at least one character of that type, which allows for O(1) == comparisons between strings of
+different types. [done!]
+
+Having a mixed string type, for large strings where you may have occasional BMP or non-BMP characters, but the major portion is ASCII/Latin1.
+   -> Keep a table of offsets and types, 2 bits per 64 character chunk for encoded type of chunk,
+      offset table, 16-bits, 32-bits, or 64-bits?
+      16-bits could handle 16+5-2 = 2^19 characters, max cost 16K for 512K character string,
+      32-bits could handle up to 32GB character strings
+
+Have a function that given a Vector{UInt8}, BinaryStr, UTF8Str, String, etc.
+returns a vector of substring'ed UniStr, which only has to make copies if some of the lines
+are _Latin, _UCS2, _UTF32.
+(This could be done as well for input of Vector{UInt16}, RawWordStr)
+
+To save space, and for better performance, lines with different types are pooled together,
+and lines with ASCII can point to the original, except if the input is Vector{UInt8},
+in which case they are also added to the pooled buffer for the lines.
+In addition, depending on the percentage of characters still pointing into the original string
+(again, only if it's not a Vector, which must be copied), it may decide to go ahead and copy
+all of the ASCII substrings into the pool.
+Note: for good substring performance, some of the operations that are optimized to work 8 bytes
+(or more) at a time, will need to deal with masking the initial chunk, not just the final chunk.
+
+
+New ideas:
+Have a single concrete "UniStr" type, which uses bits in the trailing "nul" byte of the String
+representation, to store the following information:
+
+NotValidated, Invalid, NoASCII, Latin, BMP, UTF32, Hash present, Short
+
+2 bits: 00 -> Valid, 01 -> Invalid, 10 -> NotValidated, 11 means?
+1 bit:  0  -> Some ASCII, 1 -> no ASCII (bit flipped from others so that 0 -> ASCIIStr)
+1 bit:  0  -> No Latin1,  1 -> some Latin1
+1 bit:  0  -> ByteWise,   1 -> WordWise
+1 bit:  0/1 Hash present
+1 bit:  0/1 Short
+1 bit:  ?
+
+Extra byte for wordwise:
+1 bit:  0  -> None > 0x7ff, 1 -> some > 0x7ff (for UTF8 conversions?)
+1 bit:  0  -> No BMP,   1 -> some BMP (0x800-0xd7ff,0xe000-0xffff)
+1 bit:  0  -> Only BMP, 1 -> some non-BMP (0x10000-0x10ffff)
+Have at least 5 bits for other information.
+
+So: ASCIIStr would be: Valid, All ASCII, ... i.e. 0 + short/hash bits
+    _LatinStr would be: Valid, maybe no ascii, Latin1, no bmp, no non-bmp
+    _UCS2Str  would be: Valid, maybe no ascii, maybe Latin1, some bmp, no non-bmp
+    _UTF32Str would be: Valid, maybe no ascii, maybe Latin1, maybe BMP, some non-bmp

README.md

@@ -6,13 +6,35 @@
 
 [![codecov.io](http://codecov.io/github/JuliaString/Strs.jl/coverage.svg?branch=master)](http://codecov.io/github/JuliaString/Strs.jl?branch=master)
 
+[![](https://twitter.com/twitter/statuses/995385258332901377)]
+
 The `Strs` package is now working on both the release version (v0.6.2) and the latest master (v0.7.0-DEV).
+
 It represents an attempt to give Julia better string handling than possible with Base `String` and `Char`.
 
 I am now trying to make sure that all of the functionality in String and Char is implemented for
 Str and Chr, and to start optimizing the functions (although they are already substantially faster)
 
-I also am working on implementing full Regex support (although some changes might be needed in Base to make it work with the `r"..."` regex string macro and `Regex` type, because there are some fields missing that would be needed to handle arbitrary abstract string types).
+Strs.jl is now a container for a number of different packages from [JuliaString.org](https://juliastring.org)
+
+* [StrAPI](https://github.com/JuliaString/StrAPI.jl): Common API for string/character functionality
+* [CharSetEncodings](https://github.com/JuliaString/CharSetEncodings.jl): Basic types/support for Character Sets, Encodings, and Character Set Encodings
+* [Chars](https://github.com/JuliaString/Chars.jl): `Chr{CharSet,CodeUnitType}` type and support
+* [StrBase](https://github.com/JuliaString/StrBase.jl): `Str{CSE, Hash, SubSet, Cache}` type
+* [PCRE2](https://github.com/JuliaString/PCRE2.jl): `PCRE2` library support
+* [StrRegex](https://github.com/JuliaString/StrRegex.jl): `Regex` support for all string types
+* [StrLiterals](https://github.com/JuliaString/StrLiterals.jl): Extensible string literal support
+* [Format](https://github.com/JuliaString/Format.jl): Python/C style formatting (based on [Formatting](https://github.com/JuliaIO/Formatting.jl)
+* [StrFormat](https://github.com/JuliaString/StrFormat.jl): formatting extensions for literals
+* [StrTables](https://github.com/JuliaString/StrTables.jl): low-level support for entity tables
+* [HTML_Entities](https://github.com/JuliaString/HTML_Entities.jl)
+* [LaTeX_Entities](https://github.com/JuliaString/LaTeX_Entities.jl)
+* [Emoji_Entities](https://github.com/JuliaString/Emoji_Entities.jl)
+* [Unicode_Entities](https://github.com/JuliaString/Unicode_Entities.jl)
+* [StrEntities](https://github.com/JuliaString/StrEntities.jl): Entity extensions for literals
+* [InternedStrings](https://github.com/JuliaString/InternedStrings.jl): save space by interning strings (by @oxinabox!)
+
+The new package [APITools](https://github.com/JuliaString/APITools.jl) is used to set up a consistent and easy to use API for most of the cooperating packages, without having to worry too much about imports, exports, using, and what functions are part of a public API, and which ones are part of the internal development API for other packages to extend.
 
 I would very much appreciate any constructive criticism, help implementing some of the ideas, ideas on how to make it perform better, bikeshedding on names and API, etc.
 Also, I'd love contributions of benchmark code and/or samples for different use cases of strings,
@@ -85,12 +107,13 @@ There is a new API that I am working on for indexing and searching, (however the
 
 Also there are more readable function names that always separate words with `_`, and avoid hard to understand abbreviations:
 
-* `is*`              -> `is_*` (for `ascii`, `digit`, `space`, `alpha`, `numeric`, `valid`,
+* `is*`              -> `is_*` (for `ascii`, `digit`, `space`, `numeric`, `valid`,
 		                  `defined`, `empty`, `assigned`)
 * `iscntrl`          -> `is_control`
 * `isgraph`          -> `is_graphic`
 * `isprint`          -> `is_printable`
 * `ispunct`          -> `is_punctuation`
+* `isalpha`          -> `is_letter`
 * `isalnum`          -> `is_alphanumeric`
 * `isgraphemebreak`  -> `is_grapheme_break`
 * `isgraphemebreak!` -> `is_grapheme_break!`
@@ -100,3 +123,30 @@ Also there are more readable function names that always separate words with `_`,
 * `uppercasefirst`   -> `uppercase_first`
 * `startswith`       -> `starts_with`
 * `endswith`         -> `ends_with`
+
+* In addition, I've added `is_alphabetic`
+
+## Kudos
+
+Nobody is an island, and to achieve great things, one must stand on the shoulders of giants.
+
+I would like to thank some of those giants in particular:
+
+* The four co-creators of Julia: [Jeff Bezanson](https://github.com/JeffBezanson),[Viral B. Shah](https://github.com/ViralBShah), [Alan Edelman](https://github.com/alanedelman), and [Stefan Karpinski](https://github.com/StefanKarpinski), without their uncompromising greediness, none of this would be possible.
+
+* [Tom Breloff](https://github.com/tbreloff), for showing how an ecosystem could be created in Julia, i.e. "Build it, and they will come", for providing some nice code in this [PR](https://github.com/JuliaIO/Formatting.jl/pull/10) (which I shamelessly pirated in order to create [Format](https://github.com/JuliaString/Format.jl), and for good advice at JuliaCon.
+* [Ismael Venegas Castelló](https://twitter.com/SalchiPapa1337) for encouraging me to [tweet](https://twitter.com/GandalfSoftware) about Julia starting at the 2015 JuliaCon, for good advice, and being a great guy in general.
+* [Chris Rackaukas](https://github.com/ChrisRackauckas) simply a star in Julia now, great guy, great advice, and great blogs about stuff that's usually way over my head. Julia is incredibly lucky to have him.
+* [Jacob Quinn](https://github.com/quinnj), for collaborating & discussions early on in [Strings](https://github.com/quinnj/Strings.jl) on ideas for better string support in Julia, as well as a lot of hard work on things dear to me, such as databases and importing/exporting data [SQLite](https://github.com/JuliaDatabases/SQLite.jl), [ODBC](https://github.com/JuliaDatabases/ODBC.jl), [CSV](https://github.com/JuliaData/CSV.jl), [WeakRefStrings](https://github.com/JuliaData/WeakRefStrings.jl), [DataStreams](https://github.com/JuliaData/DataStreams.jl), [Feather](https://github.com/JuliaData/Feather.jl), [JSON2](https://github.com/quinnj/JSON2.jl)
+* [Milan Bouchet-Valat](https://github.com/nalimilan), for discussions on string handling and encoding in [StringEncodings](https://github.com/nalimilan/StringEncodings.jl)
+* [Tim Holy](https://github.com/timholy) for the famous "Holy" Trait Trick, which I use extensively in the Str* packages, for the work along with [Matt Bauman](https://github.com/mbauman) on making Julia arrays general, extensible while still performing well, and hence very useful in my work.
+* [Steven G. Johnson]() for illuminating me on how one could create a whole package in very few lines of code when I first started learning Julia, see [DecFP](https://github.com/stevengj/DecFP.jl)
+* [Tony Kelman](https://github.com/tkelman), for very thorough reviews of my PRs, I learned a great deal from his (and other Julians') comments), including that I didn't have to code in C anymore to get the performance I desired.
+
+* [Lyndon White](https://github.com/oxinabox), I've already "appropriated" :grinning: his very nice [InternedStrings](https://github.com/JuliaString/InternedStrings.jl) into this package, I'm really lucky to have gotten him to join the organization!
+* [Bogumił Kamiński](https://github.com/bkamins) who has been doing a great job testing and reviewing `Strs` (as well as doing the same for the string/character support in Julia Base), as well as input into the design. (Also very glad to have co-opted him to become a member of the org)
+* Last but not least, Julia mathematical artist (and blogger!) extraordinaire, [Cormullion](https://github.com/cormullion), creator of our wonderful logo!
+
+Also thanks to anybody who's answered my (sometimes stupid :grinning:) questions on [Gitter](https://gitter.im/JuliaLang/julia) and [Discourse](https://discourse.julialang.org/)
+
+Kudos to all of the other contributors to Julia and the ever expanding Julia ecosystem!

REQUIRE

@@ -1,2 +1,5 @@
 julia 0.6
-CharSetEncodings
+StrRegex
+StrFormat
+StrEntities
+InternedStrings