New flattables_sample with stock exchange end-of-week data

The previous flattables_sample was three tables of Mickey Mouse data. Actually, Wombat and Flintstones stuff.

I think people find it challenging to come up with a use-case for flattables. And having multiple tables may have given the impression that it's a kind of in-memory database. It's true that gotables can be used as a flexible in-memory data store, and can easily save or transmit data. However, flattables is more of a create-a-table-and-transmit-it kind of format. It is a fast binary constrained (tables only) FlatBuffers that takes the effort out of designing and writing the schema and plumbing code, so long as you're happy with tabula data.

The new flattables_sample is an end-of-week dump of data from the Australian Stock Exchange. It conforms perfectly with the flattables tabular format. And you can, of course, add one or more other tables to the transmission to carry, say, metadata about the main payload.

Some minor cosmetic bug fixes have been made. No actual consequential bugs, that I'm aware of.

Consistent with the it's-not-a-database theme, I went through and surgically removed all the GraphQL code. It seemed like a fine idea at the time, but that's not what flattables (or FlatBuffers) is, and so it had to go.

The flattables_sample table is around 1,500 rows. Here are the first few lines to give you and idea:

[AsxMarketDataWeek20190712]
ticker     date      open      high       low     close   volume
string    int32   float32   float32   float32   float32    int32
"360"  20190712     3.8       3.8       3.8       3.8     207325
"14D"  20190712     0.29      0.29      0.28      0.285   173125
"1AD"  20190712     0.17      0.17      0.17      0.17     25000
"1ST"  20190712     0.093     0.098     0.091     0.097   969749
"2BE"  20190712     0.395     0.395     0.345     0.36    384609
"3DP"  20190712     0.046     0.047     0.045     0.047   545265
"3PL"  20190712     0.967     0.967     0.965     0.965   243734
"4CE"  20190712     0.02      0.02      0.016     0.018  3725883
"4DS"  20190712     0.061     0.061     0.058     0.058  1845766
"5GN"  20190712     1.62      1.62      1.55      1.58    296411
"88E"  20190712     0.018     0.018     0.017     0.018  2317656
"8CO"  20190712     0.16      0.16      0.16      0.16      6500
"8EC"  20190712     0.8       0.8       0.8       0.8        991
"8IH"  20190712     0.06      0.06      0.06      0.06      3500
"9SP"  20190712     0.023     0.024     0.023     0.023 10147822
"A1G"  20190712     0.16      0.16      0.16      0.16      2300
"A1M"  20190712     0.37      0.42      0.37      0.42     73776
"A2B"  20190712     1.715     1.785     1.715     1.78     38280
"A2M"  20190712    15.81     15.94     15.79     15.85   2081197
"A3D"  20190712     0.36      0.385     0.36      0.365    74680
"A40"  20190712     0.13      0.135     0.13      0.135  1704143
"A4N"  20190712     0.115     0.115     0.115     0.115    15000
"AAA"  20190712    50.09     50.1      50.09     50.09    629689

Three fast functions for reading/writing FlatBuffers

Substantial performance improvements in functions and methods used with reading and writing FlatBuffers.

Sample code can be found here with Examples of new functions for processing FlatBuffers that are much faster.

https://godoc.org/github.com/urban-wombat/flattables_sample#pkg-examples

The new functions are:

• NewFlatBuffersFromSlice()
• NewSliceFromFlatBuffers()
• OldSliceFromFlatBuffers()

All three functions move data between FlatBuffers and a simple Go collection of tables.

OldSliceFromFlatBuffers() reuses memory which reduces cpu and memory consumption.

To get started with gotables FlatBuffers see https://github.com/urban-wombat/flattables#a-simple-and-fast-way-to-get-started-with-google-flatbuffers

It may sound hard to believe, but all the code at https://github.com/urban-wombat/flattables_sample was auto-generated in seconds a single gotables schema file.

The objective is to define your FlatBuffers table(s) and run flattablesc to generate both Google FlatBuffers flatc code and specialised gotables table-only code.

Google FlatTables support: more methods and documentation

In a nutshell:

• Install FlatTables
• Write a data tables file
• Run $ gotflat -f ../flattables_sample/tables.got -n flattables_sample -p github.com/urban-wombat/flattables_sample
• You get this: FlatBuffers API
• And this: FlatBuffers Tutorial

Recent work has focused on Google FlatTables support.

More methods. More documentation.

• gotables now provides a broad introduction to FlatTables, the gotables support library for FlatBuffers.

• flattables shows how to install gotables, flattables and the associated utility gotflat which generates Go code and a project-specific tutorial explaining how to use FlatBuffers with the FlatTables library.

• flattables_sample_main.go is a project-specific tutorial generated from my sample gotables file tables.got.

If I've missed something in the instructions that would help you get started, hit me up on: urban.wombat.burrow@gmail.com

flatbuffers schema can now have a (deprecated) attribute.

To deprecate a gotables column (flatbuffers field) in your gotables
file you are using as a schema and code generator source:

rename the column and append _deprecated_ to the name.

myColumnName becomes myColumnName_deprecated_

All data must be sent using the original name myColumnName (from processes
that don't know about the deprecation change), or must send no data at all
for myColumnName.

See previous releases (below) for how to install and use the flattables flatbuffers code generation.

Refer to flatbuffers Writing A Schema
to understand some of the benefits and gotchas of deprecated fields.

Additional functions

New functions

• NewTableSetFromFlatBuffers.go

• main.go

Note that with most other functions, these are generated specifically for your particular data tables,
similar to the way flatc generates the underlying Go functions, which the above functions call into.

An easy way to get started with Google FlatBuffers using tables.

Google FlatBuffers is incredibly fast.

This library of Go templates takes a simple gotables data file and generates a conforming Google FlatBuffers
schema file.

gotft takes this schema file
and calls the Google FlatBuffers utility flatc, and calls into the flattables library to generate helper
functions to write and read FlatBuffers []byte arrays. It also generates tests and bench tests which you can run.

It provides a major simplification of FlatBuffers use to tabular tables. Which means your objects need to be
normalised sufficiently to write to tables, and be capable of denormalisation at the receiving end.

The code generation (using gotft - and flatc under the covers) is blindingly simple and easy.

The repository https://github.com/urban-wombat/flattables_sample was generated from scratch
from just one gotables data file: https://github.com/urban-wombat/flattables_sample/blob/master/tables.got
by running the following command:

gotft -f tables.got -n flattables_sample

And here's what you can run straight after that code generation step:

$ go test -bench=.
goos: windows
goarch: amd64
pkg: github.com/urban-wombat/flattables_sample
BenchmarkGetFlatBuffersAndCompareWithGotables-4           200000              9006 ns/op
BenchmarkGetFlatBuffersOnly-4                            5000000               269 ns/op
PASS
ok      github.com/urban-wombat/flattables_sample       3.603s