v3.27.0 cherry pick from private #2204

kordless · 2023-01-12T15:22:31Z

This is (hopefully) the last release before we merge repos. 🚀

(cherry picked from commit d33bf48)

…nfo (#2336) * WIP: Convert SchemaAPI to be DAX-centric * Tables(), CreateField() * CreateTable(), DeleteTable(), DeleteField() * More cleanup * Remove the old SchemaAPI (cherry picked from commit a15783c)

…2338) * This adds in support to the lattice UI application to use the SQL3 endpoint. If the `/sql` endpoint returns 404, it will use the SQL1 endpoint. If the `/sql` endpoint is available, it will send SQL queries to that. It does not try the SQL1 endpoint if the SQL3 endpoint returns an error processing the query. That is, it is all SQL3 or SQL1. - Below are the specific changes: - Adds a file that contains functions for interacting with http services as opposed to just grpc/event-based services. As of this commit, it is only the SQL3 endpoint. - This adds a variable to track if we are using the SQL3 endpoint or not - This adds a function to handle the response from the SQL3 endpoint - Adds a function to eventServices to query the sql3 HTTP endpoint - Fixed a missing semicolon in grpcServices Co-authored-by: Fletcher Haynes <fletcher.haynes@generalassemb.ly> (cherry picked from commit 7ab453e)

…ing)set map type conversions (#2342) (cherry picked from commit 728b1dc)

substring(string, startIndex,length). (cherry picked from commit aa2a62f)

…hemaAPI (#2340) * Thread Table.Description through SchemaAPI * Thread Table.CreatedAt through SchemaAPI * Thread Field.CreatedAt through SchemaAPI (cherry picked from commit 734477a)

* Fb:1787 - Clean up * fb-1787 : String upper function * Formatting the files (cherry picked from commit 57ce7c4)

* Move batch.Importer interface to pilosa.Importer In addition to moving the interface, it updates all the methods to use dax.TableID (for example) intead of a string pilosa index name. * Change unused onPremImporter methods to no-op. onPremImporter is a wrapper around API which implements the Importer interface. This is currently only used by sql3 running locally in standard (i.e not "serverless") mode. Because sql3 always sets `useShardTransactionalEndpoint = true`, There are several methods which this implemtation of the Importer interface does not use, and therefore they intentionally no-op. (cherry picked from commit 12d608c)

#2348) (cherry picked from commit 5110405)

) This commit introduces an interface called `TableKeyer` which anything that means to represent a "table" can implement. Examples are `dax.QualifiedTable`, `dax.Table`, and `string` (for legacy pilosa calls where Execute simply took `index string`). In the case of `orchestrator.Execute()` and `qualifiedOrchestrator.Execute()`, we are intentionally strict about which type of `TableKeyer` the respective method accepts. If we find, in the future, this is too restrictive, we can loosen that; but for now it helps us understand what is expected. (cherry picked from commit a61d1a9)

(cherry picked from commit 9548e71)

* Fix "qualifer" misspellings * Remove `track_existence` and `shard_width` from SHOW TABLES output * Thread Owner, UpdatedAt, UpdatedBy through SchemaAPI I took the liberty of renaming "LastUpdatedUser" to "UpdateBy" to align with "UpdatedAt". (cherry picked from commit 63cfdb5)

* Make interfaces more specific than "MDS" - Introduce `dax.Schemar` interface - Introduce `dax.Noder` interface - The rest is generally to standardize on the new interfaces. - Remove `pilosa.SchemaInfoAPI` interface - Move `TranslateNode` and `ComputeNode` types from controller to dax package - Remove `queryer.FeatureBaseImporter` - Remove `queryer.MDS` interface - Remove `queryer.Importer` interface - Identify types using an "MDS" interface and split into Noder/Schemar as necessary - Changed `Queryer.orchestrator` to a `map[qual]*qualifiedOrchestrator` because we can't share an orchestrator across quals * Convert orchestrator to use TableKeyer (cherry picked from commit 14f1930)

* fb-1802 : Trim functionality rebase * fb-1802 : trim - updated as per review (cherry picked from commit ac3ffac)

Embeded dataframe-csv-loader command as featurebase subservice (cherry picked from commit f2a13c8)

) (cherry picked from commit e572c8f)

(cherry picked from commit 2146f40)

* Prevent file corruption when writing tar backup to stdout FB-1794 Tar backups written to stdout were coming out corrupt. This turned out to be due to log messages getting written to stdout and ending up in the tar file. We now check to see if the tar file and the log are both going to stdout, and if they are, send the logs to stderr instead. Testing did not have any kind of consistency or validity check. We now compare a tar file sent to a file and a tar file sent to stdout to make sure they're the same. This does not guarantee correctness but does at least catch this form of corruption. * trying different index name Co-authored-by: tgruben <tgruben@gmail.com> Co-authored-by: Todd Gruben <todd@molecula.com> (cherry picked from commit a8996a1)

* Implemented SQL3 LOWER() (cherry picked from commit 4773aab)

(cherry picked from commit eca3168)

(cherry picked from commit 9759602)

(cherry picked from commit c1dbc48)

(cherry picked from commit a4f18fb)

(cherry picked from commit 690a937)

(cherry picked from commit 35c472a)

(cherry picked from commit bbaa7dd)

(cherry picked from commit dbb6d53)

- check that serverlessStorage is not nil before closing it - check that we don't already hold a lock on a serverless storage Manager before trying to load it. This fixed at least one test failure. (cherry picked from commit 87d1c31)

(cherry picked from commit 797b8bc)

changed empty snapshots/writelogs to return nil which was causing NPE (cherry picked from commit bee666c)

* distribute bulk insert to owning node (cherry picked from commit e8505d8)

…_time (#2394) * Change JSON response name from exec_time to execution-time Execution time stopped working in the CLI because it uses the latest json tag. * Wait, don't break the interface. * Add a test for the sql response json tags. This is to make sure that if someone like Travis just goes and changes a tag name to be more consistent, that we perhaps catch that before it gets to the end user. * Change exec_time to execution-time after all (cherry picked from commit b5dd3ea)

* check root operator after optimize * round of bug fixes (cherry picked from commit 9f04221)

added space_used columns to show tables and fb_cluster_nodes (cherry picked from commit 164aac5)

* adding more error handling for rbf Co-authored-by: Jacob Brinlee <jacobbrinlee@Jacobs-MBP.attlocal.net> (cherry picked from commit c9ce26c)

…8) (#2398) * handle insert into timequantum fields with default 'now' time * allocate on the stack (cherry picked from commit 18fe6a3)

FB-1861 (cherry picked from commit da74f0a)

(cherry picked from commit 4750c22)

Co-authored-by: Julio <julio.martinez@featurebase.com> (cherry picked from commit 581a4d1)

…(#2402) Co-authored-by: Julio <julio.martinez@featurebase.com> (cherry picked from commit 3f91cc3)

* FB-1814: Implement ASCII() (cherry picked from commit f590227)

* handle sum aggregates with ints; handle escaped quotes in blob literals * added test ceoverage * skip subquery test for dax (cherry picked from commit af475a2)

Co-authored-by: Kord Campbell <kord@Bob.local> Co-authored-by: Fletcher Haynes <fletcher@capitalprawn.com> (cherry picked from commit 17cb2cb)

CLAassistant · 2023-01-12T15:22:43Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 14 committers have signed the CLA.

✅ pokeeffe-molecula
✅ ch7ck
❌ rachithrr
❌ travisturner
❌ HHans09
❌ fhaynes
❌ jaffee
❌ bruce-b-molecula
❌ lorycloutier
❌ tgruben
❌ Kord Campbell
❌ jrbrinlee1
❌ kordless
❌ j3m7

Kord Campbell seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

sonarqubecloud · 2023-01-12T20:34:31Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
122 Code Smells

No Coverage information
11.5% Duplication

rachithrr and others added 30 commits January 10, 2023 23:16

FB-1795: Implement REVERSE() scalar string function (#2335)

a74b5c7

(cherry picked from commit d33bf48)

Convert SchemaAPI interface to use dax.Table instead of pilosa.IndexI…

0985eeb

…nfo (#2336) * WIP: Convert SchemaAPI to be DAX-centric * Tables(), CreateField() * CreateTable(), DeleteTable(), DeleteField() * More cleanup * Remove the old SchemaAPI (cherry picked from commit a15783c)

handle int-->bool map type conversions; handle single value-->(id|str…

eb54530

…ing)set map type conversions (#2342) (cherry picked from commit 728b1dc)

FB-1800: Implement SUBSTRING() (#2343)

c39e599

substring(string, startIndex,length). (cherry picked from commit aa2a62f)

Add Table.Description, Table.CreatedAt, Field.CreatedAt support to Sc…

c453212

…hemaAPI (#2340) * Thread Table.Description through SchemaAPI * Thread Table.CreatedAt through SchemaAPI * Thread Field.CreatedAt through SchemaAPI (cherry picked from commit 734477a)

Fb:1787 - Clean up (#2339)

9346e26

* Fb:1787 - Clean up * fb-1787 : String upper function * Formatting the files (cherry picked from commit 57ce7c4)

Remove trackExistence check in batch (i.e. always build _exists) data. (

0cb16f0

#2348) (cherry picked from commit 5110405)

FB-1805: implement REPLACEALL() (#2349)

17188b7

(cherry picked from commit 9548e71)

fb-1802 : Trim functionality (#2353)

80626df

* fb-1802 : Trim functionality rebase * fb-1802 : trim - updated as per review (cherry picked from commit ac3ffac)

Fb 1874 dataframe-csv-loader featurebase subcommand (#2341)

a2e14df

Embeded dataframe-csv-loader command as featurebase subservice (cherry picked from commit f2a13c8)

Fix dax docker-compose (dc-up) which was broken by ServiceManager (#2356

9e2dbad

) (cherry picked from commit e572c8f)

fixed csv bugs (#2355)

f2442dc

(cherry picked from commit 2146f40)

FB-1719: implement SQL3 lower() (#2358)

838bc2d

* Implemented SQL3 LOWER() (cherry picked from commit 4773aab)

implement having; create view experiment (#2357)

7599941

(cherry picked from commit eca3168)

added updated_at column to show tables output (#2364)

b46a82e

(cherry picked from commit 9759602)

fb-1809: SQL3 RTrim & LTrim func for strings (#2361)

766db38

(cherry picked from commit c1dbc48)

FB-1812: implement stringsplit() (#2362)

e8ac1a0

(cherry picked from commit a4f18fb)

remove alpha director (unused)

71e3c00

(cherry picked from commit 690a937)

extremely WIP

ed7c6d4

(cherry picked from commit 35c472a)

more WIP

ff3595d

(cherry picked from commit bbaa7dd)

implement closing on dax, remove all locks when shutting down

7668275

(cherry picked from commit dbb6d53)

several fixes and debug logging

6cef68a

- check that serverlessStorage is not nil before closing it - check that we don't already hold a lock on a serverless storage Manager before trying to load it. This fixed at least one test failure. (cherry picked from commit 87d1c31)

fix lint

bc3123d

(cherry picked from commit 797b8bc)

fix dumb issue on storage manager test

4d678fa

changed empty snapshots/writelogs to return nil which was causing NPE (cherry picked from commit bee666c)

tgruben and others added 19 commits January 10, 2023 23:28

[FB-1831] distribute bulk insert to owning node (#2391)

5ec31d4

* distribute bulk insert to owning node (cherry picked from commit e8505d8)

Bug fix round up (fb-1841, fb-1819, fb-1867) (#2396)

4e43b76

* check root operator after optimize * round of bug fixes (cherry picked from commit 9f04221)

added space_used columns (#2397)

a6317c5

added space_used columns to show tables and fb_cluster_nodes (cherry picked from commit 164aac5)

adding more error handling for rbf (#2395)

3b233d2

* adding more error handling for rbf Co-authored-by: Jacob Brinlee <jacobbrinlee@Jacobs-MBP.attlocal.net> (cherry picked from commit c9ce26c)

handle insert into timequantum fields with default 'now' time (fb-186…

5e151ee

…8) (#2398) * handle insert into timequantum fields with default 'now' time * allocate on the stack (cherry picked from commit 18fe6a3)

Implement SPACE() function for SQL3 (#2399)

2b9c134

FB-1861 (cherry picked from commit da74f0a)

include space_used column in show table output (#2401)

3b1707c

(cherry picked from commit 4750c22)

fix import paths

d22bd94

fix indent

77305b5

fix import paths and import cycles

7acf326

Add dd tracer and associated config options (#2403)

c236085

Co-authored-by: Julio <julio.martinez@featurebase.com> (cherry picked from commit 581a4d1)

Create main log with 640 permissions, leave all other logs the same. …

9030a05

…(#2402) Co-authored-by: Julio <julio.martinez@featurebase.com> (cherry picked from commit 3f91cc3)

FB-1814: Implement ASCII() (#2378)

1cdcaf4

* FB-1814: Implement ASCII() (cherry picked from commit f590227)

fix sum aggregate (fb-1874) (#2404)

0ddf69a

* handle sum aggregates with ints; handle escaped quotes in blob literals * added test ceoverage * skip subquery test for dax (cherry picked from commit af475a2)

fixes FB-1873 (#2400)

e2c3b40

Co-authored-by: Kord Campbell <kord@Bob.local> Co-authored-by: Fletcher Haynes <fletcher@capitalprawn.com> (cherry picked from commit 17cb2cb)

sync cherry-picks 4750c22..17cb2cb

fee6d85

sync cherry-picks 4750c22..17cb2cb

4817806

add LICENSE

3a7d6ef

kordless requested a review from ch7ck January 12, 2023 15:22

ch7ck force-pushed the 1-10-2023-cherry-pick branch 3 times, most recently from e4b00a2 to 033dd72 Compare January 12, 2023 20:17

resolve go vet errors related to redeclares

475bf58

ch7ck force-pushed the 1-10-2023-cherry-pick branch from 033dd72 to 475bf58 Compare January 12, 2023 20:33

ch7ck approved these changes Jan 12, 2023

View reviewed changes

ch7ck merged commit 5274cae into master Jan 12, 2023

ch7ck deleted the 1-10-2023-cherry-pick branch January 12, 2023 22:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

v3.27.0 cherry pick from private #2204

v3.27.0 cherry pick from private #2204

Uh oh!

kordless commented Jan 12, 2023

Uh oh!

CLAassistant commented Jan 12, 2023 •

edited

Loading

Uh oh!

sonarqubecloud bot commented Jan 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

v3.27.0 cherry pick from private #2204

v3.27.0 cherry pick from private #2204

Uh oh!

Conversation

kordless commented Jan 12, 2023

Uh oh!

CLAassistant commented Jan 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Jan 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

CLAassistant commented Jan 12, 2023 •

edited

Loading