Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fireant 1.0 Rewrite #151

Merged
merged 163 commits into from
Sep 14, 2018
Merged
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
163 commits
Select commit Hold shift + click to select a range
2430f79
Initial commit of 1.0
twheys Jan 19, 2018
6bc009e
Added highcharts transformer
twheys Jan 23, 2018
d590d18
Added functionality for operations, added CumSum, CumAvg, and CumProd
twheys Jan 24, 2018
7f2dc5a
Added support for Totals for MySQL and Vertica backends
twheys Jan 26, 2018
32d2e20
Added fix for aligning weekdays through leap years when querying YoY …
twheys Jan 26, 2018
13fff75
Cleaned up, addressed PR comments, and changed the highcharts transfo…
twheys Jan 26, 2018
f1b1c7f
Added an entry point for querying dimension options
twheys Jan 29, 2018
4fad225
Added support for pagination through limit and offset parameters
twheys Jan 29, 2018
2a1d21d
Bumped pypika version in setup.py
twheys Jan 29, 2018
a7a5530
Polish and fixes for 1.0.0
twheys Feb 1, 2018
c7d8b0b
Added pandas transformer
twheys Feb 1, 2018
af32d03
Added CSV transformer
twheys Feb 1, 2018
81c9abd
Fixed some issues reported by codacy
twheys Feb 2, 2018
98ad44d
Added more tests
twheys Feb 5, 2018
11a7fab
fixes for totals, colors, etc issues found during manual testing
twheys Feb 8, 2018
14eff1d
Added formatting to highcharts
twheys Feb 8, 2018
2c3aeda
bumped dev version
twheys Feb 8, 2018
d74ffe3
- Fixed references to share a query if they're for the same dimension…
twheys Feb 9, 2018
8fc1599
added fucking pie charts
twheys Feb 9, 2018
06ba987
bumped dev version to 4
twheys Feb 9, 2018
da1ee22
fixed the responses to format infinity
twheys Feb 9, 2018
5c07ff7
Fixed an issue when querying display options for a unique dimension w…
twheys Feb 12, 2018
addc42a
Bumped version to 1.0.0.dev6
twheys Feb 12, 2018
ec3f048
Fixed the display options query to work for categorical dimensions ag…
twheys Feb 12, 2018
0a9b19b
Bumped version to 1.0.0.dev7
twheys Feb 12, 2018
91ee1af
Fixed area percentage charts to using percent stacking
twheys Feb 12, 2018
23a365e
Fixed the even worse bug caused by the last commit that changed how d…
twheys Feb 12, 2018
d24bc65
Removed unused import in vertica
twheys Feb 12, 2018
255fba7
Fixed a bug where dimension values were not cast to string and bumped…
twheys Feb 14, 2018
daa9145
Fixed an issue with toposort where it would try to sort joins when th…
twheys Feb 15, 2018
bb3a2f7
Incremented version to 1.0.0.dev10
twheys Feb 15, 2018
fb073d9
Fixed the way the data frame is formatted given an int column with nu…
twheys Feb 16, 2018
2e7a99f
changed the formatting of infinity to return None
twheys Feb 23, 2018
562f6ad
bumped version to dev12
twheys Feb 23, 2018
7af181e
Some bug fixes
twheys Feb 26, 2018
6914fe1
Added better support for totals queries by using UNIONs instead of RO…
twheys Mar 6, 2018
18b644c
Bumped version to 1.0.0.dev14
twheys Mar 6, 2018
2c2e9d8
Fixed references to both work without selecting the referenced dimens…
twheys Mar 7, 2018
d50d255
Bumped pypika dependency to 0.10.6 and version to dev16
twheys Mar 7, 2018
09853f6
Fixed the setup.py pypika dependency
twheys Mar 7, 2018
73c1b21
Added slow query warning log for queries over 15s
mikeengland Mar 8, 2018
04908ae
Moved slow query log min duration constant to be an attribute of data…
mikeengland Mar 9, 2018
5906ebb
Lower cased the slow query log attribute
mikeengland Mar 9, 2018
f1f6db1
Merge pull request #152 from mikeengland/fireant1.0
mikeengland Mar 9, 2018
7e65fb7
Bump version to 1.0.0.dev17
mikeengland Mar 9, 2018
b1dbb75
Added NOT LIKE support and renamed WildcardFilter to LikeFilter
mikeengland Mar 9, 2018
dd7ed89
Merge pull request #154 from mikeengland/add-negated-wildcard-filter
mikeengland Mar 9, 2018
cabfc95
Bumped version to 1.0.0.dev18
mikeengland Mar 9, 2018
79776cf
Added stacked area charts
twheys Mar 12, 2018
9cca39e
Merge pull request #155 from kayak/stacked_area_chart
mikeengland Mar 12, 2018
15d8442
bumped version to dev19
twheys Mar 12, 2018
14dfeac
Added options to highcharts transformers to enable/disable axis label…
twheys Mar 19, 2018
8450caf
bumped version to dev20
twheys Mar 19, 2018
8d07093
Fixes #105 Added a VerticaQueryBuilder class with support for a query…
twheys Mar 22, 2018
fd56d94
added kwarg to fetch which allows a query hint to be used for support…
twheys Mar 23, 2018
bf27790
bumped version to dev21
twheys Mar 23, 2018
e620d11
Fixed the hint query to prevent it from being used unless it actually…
twheys Mar 23, 2018
e64d001
bumped to dev22 and updated reqs in setup.py
twheys Mar 23, 2018
0d7847a
Changed the API of highcharts to include a series level in between ax…
twheys Apr 10, 2018
da79bbd
bumped to dev version dev23
twheys Apr 10, 2018
4e3e093
Fixed 'operations' in Highcharts to ignore items without operations
twheys May 31, 2018
e1a4064
bumped version to dev24
twheys May 31, 2018
1652894
Added like and not_like functions to display dimension
twheys May 31, 2018
6205d6b
bumped version to dev25
twheys May 31, 2018
ff9f16d
Fixed operations and metrics functions in highcharts transformer
twheys Jun 1, 2018
5db523e
bumped version to dev26
twheys Jun 1, 2018
2c3b915
Added a pattern dimension type
twheys Jun 26, 2018
10fd241
Merge pull request #156 from kayak/pattern_dimensions
twheys Jun 26, 2018
b787343
Bumped version to dev27
twheys Jun 26, 2018
00b93f0
added like and not_like filters to categorical dimension
twheys Jun 27, 2018
4995f84
Merge pull request #157 from kayak/category_like_filters
twheys Jun 27, 2018
2070a9c
bumped version to dev28
twheys Jun 27, 2018
1fa41c6
Fixed the pattern dimension
twheys Jun 28, 2018
15a2846
fixing codacy issues
twheys Jun 28, 2018
5c74af3
Updated version to dev29
twheys Jun 28, 2018
cb16918
Changed like and not_like filters to support multiple patterns
twheys Jul 3, 2018
9d9143a
Merge pull request #158 from kayak/multipattern_filters
twheys Jul 3, 2018
09b6264
bumped version to dev30
twheys Jul 3, 2018
ceee6bb
Fixed coerce type to allow literal NAN and INF strings
twheys Jul 3, 2018
71f8f56
bumped version to dev31
twheys Jul 3, 2018
3bd0f98
Fixes for the generated aliases, prepending $ to make them distinct
twheys Jul 5, 2018
2356429
Merge pull request #159 from kayak/fix_aliases
twheys Jul 5, 2018
9aef44b
Bumped version to 1.0.0.dev32
twheys Jul 5, 2018
35e24d5
Fixed dimension choices query for unique dimensions with display defi…
twheys Jul 5, 2018
b2befce
Merge pull request #160 from kayak/fixed_dimension_choices_query
twheys Jul 5, 2018
9f58004
Bumped version to 1.0.0.dev33
twheys Jul 5, 2018
b968a3b
Moved the limit/offset functions to the slicer query builder function…
twheys Jul 6, 2018
2406ac8
Merge pull request #161 from kayak/limit_offset_in_slicer_query_builder
twheys Jul 6, 2018
a8ff2de
Bumped version to 1.0.0.dev34
twheys Jul 6, 2018
2c8fe4e
Fixed the key used to store operations in the data query result data …
twheys Jul 6, 2018
f1209af
Bumped version to 1.0.0.dev35
twheys Jul 6, 2018
49b0368
Upgraded pypika to 0.14.7
twheys Jul 6, 2018
d661a26
Fixed the pivoted data tables widgets to render the data in the corre…
twheys Jul 9, 2018
871cd6d
Merge pull request #162 from kayak/fix_col_index_tables
mikeengland Jul 9, 2018
966710f
bumped v1.0 version to dev36
mikeengland Jul 9, 2018
19527f7
Merge pull request #163 from mikeengland/fireant1.0
mikeengland Jul 9, 2018
ef084f8
Fixed the db query alias prefix to distinguish between metrics and di…
twheys Jul 10, 2018
6941cfd
Merge pull request #164 from kayak/fix_metric_dimension_alias_collision
twheys Jul 10, 2018
1582491
Bumped version to 1.0.0.dev37
twheys Jul 10, 2018
9807d07
Added special case for using rolling operation with date dimension an…
twheys Jul 10, 2018
723d816
Merge pull request #165 from kayak/rolling_op_special_cases
twheys Jul 10, 2018
174779c
Bumped version to 1.0.0.dev38
twheys Jul 10, 2018
9cdce05
Fixed the metric format function to correctly handle for inf and nan
twheys Jul 19, 2018
89d1896
Merge pull request #166 from kayak/metric_with_zero_precision_and_inf…
twheys Jul 19, 2018
3e98dee
Fixed an issue using int values in data tables with pivot=true when t…
twheys Jul 19, 2018
15e6e8f
Fixed the behavior of antipattern to negate the whole group of patter…
twheys Jul 19, 2018
9e121be
Merge pull request #167 from kayak/fix_int_dimension_values_datatable…
twheys Jul 19, 2018
d6e295e
Merge pull request #169 from kayak/fix_antipattern_negation_fixed
twheys Jul 19, 2018
a0ee1b3
upgraded to dev39
twheys Jul 19, 2018
4cc80be
Added react-table transformer and changed pivot behavior of Pandas an…
twheys Aug 6, 2018
bdcad96
Fixed an issue with older python syntax
twheys Aug 6, 2018
46a35b1
Added a comment to the fireant transformer explaining how to use it.
twheys Aug 7, 2018
6fdd7a8
Fixed an issue with react-table when the result set is empty
twheys Aug 7, 2018
79996ef
Added docstrings to react table
twheys Aug 7, 2018
cc98225
Merge pull request #170 from kayak/reacttable-transformer
twheys Aug 7, 2018
7682e53
Bumped version to dev40
twheys Aug 7, 2018
72d1f86
Set vertica_python to replace unicode errors
twheys Aug 7, 2018
a3f6250
Merge pull request #171 from kayak/fix_unicode_errors_for_vertica
twheys Aug 7, 2018
bc11af3
Bumped version to dev41
twheys Aug 7, 2018
82b1881
Changed the pattern filters to be case insensitive
twheys Aug 7, 2018
be27a5e
Merge pull request #172 from kayak/pattern_filters_ilike
twheys Aug 7, 2018
aee5f2a
bumped fireant to dev42
twheys Aug 7, 2018
8c8f557
Fixed operations to calculate for references
twheys Aug 7, 2018
22d4030
Added a fillna to reacttable to handle NaNs and Inf
twheys Aug 7, 2018
5b9bac6
Merge pull request #174 from kayak/fix_nans_in_reacttable
twheys Aug 7, 2018
db5e1a1
bumped fireant to dev43
twheys Aug 7, 2018
ea00f81
Fixed react table's display value formatting and added test cases
twheys Aug 8, 2018
ec0f93f
Merge pull request #175 from kayak/fix_reacttable_display_value_formats
twheys Aug 8, 2018
9e134c0
fixed react tables tests
twheys Aug 8, 2018
143ce2b
bumped version to dev45
twheys Aug 8, 2018
d9976e6
Fixed reacttable to use the same formatting as datatablesjs
twheys Aug 8, 2018
186fad6
Merge pull request #176 from kayak/fix_react_formats_again
twheys Aug 8, 2018
fb4c8b7
bumped version to dev46
twheys Aug 8, 2018
c8dd245
Fixed the formatting of NaNs, Infs, negative USD values
twheys Aug 10, 2018
a28eb95
Merge pull request #177 from kayak/fix_formatting_for_negative_usd_an…
twheys Aug 10, 2018
948e95d
bumped version to dev47
twheys Aug 10, 2018
486c512
Added support for sorting by column to pandas, reacttable, and csv wi…
twheys Aug 23, 2018
35acfa6
Fixed the replacement of display values on multiindex data frames in …
twheys Aug 23, 2018
a6329e0
Remove code that would change the column accessor to data for a data …
twheys Aug 23, 2018
f3d0c74
Merge pull request #178 from kayak/pandas_sort_columns
twheys Aug 27, 2018
dfdc044
Merge pull request #179 from kayak/pandas_csv_incorrect_dimension_value
twheys Aug 27, 2018
90b9456
Bumped pypika to 0.15.2 and version to dev48
twheys Aug 27, 2018
d0ccce7
Added handle for empty sort array in pandas transformer
twheys Aug 29, 2018
029f157
Fixed react table transformer to sort with NaN values in the column b…
twheys Aug 29, 2018
c73438b
Fixed a case where if a null was returned for a display value, the hi…
twheys Aug 29, 2018
62dddb2
Merge pull request #180 from kayak/handle_empty_sort_array_pandas
twheys Aug 29, 2018
93cc081
Merge pull request #181 from kayak/fix_sorting_with_nans_pandas
twheys Aug 29, 2018
deaeb60
Merge pull request #182 from kayak/fix_nans_occurring_in_dispaly_valu…
twheys Aug 29, 2018
e82fa31
Bumped version to dev49
twheys Aug 29, 2018
1a4d872
Changed pandas transformer to filter out invalid column indices in th…
twheys Aug 30, 2018
04a7c2d
Merge pull request #183 from kayak/handle_invalid_column_idx_in_sort
twheys Aug 30, 2018
607c659
bumped version to dev50
twheys Aug 30, 2018
d45ae54
Fixed an issue where filled values in a sorted/pivoted pandas data fr…
twheys Sep 11, 2018
736ea97
Fixed another issue related to sorting data frames in the pandas tran…
twheys Sep 11, 2018
3e3fcc2
Merge pull request #184 from kayak/fix_fillna_in_pandas_sortdataframe
twheys Sep 11, 2018
ea28fe4
Added docs for 1.0.0
twheys Sep 12, 2018
2b82ae4
Added a comment to Pandas.sort_data_frame to clarify the cases when s…
twheys Sep 12, 2018
60dc5dd
Merge pull request #185 from kayak/fix_sorting_in_pandas_tx_for_no_index
twheys Sep 12, 2018
465bd31
Fixed the colors for highcharts. Added formatting to the pie chart to…
twheys Sep 12, 2018
a244eef
Merge pull request #187 from kayak/highcharts_pie_chart
twheys Sep 13, 2018
fd0c571
Added the matplotlib widget
twheys Sep 13, 2018
5ff8aa8
Removing matplotlib from travis-ci yaml and making it possible to ski…
twheys Sep 14, 2018
53301bd
Merge pull request #189 from kayak/matplotlib_widget
twheys Sep 14, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
language: python

python:
- "2.7"
- "3.3"
- "3.4"
- "3.5"
Expand Down
10 changes: 8 additions & 2 deletions fireant/__init__.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,8 @@
# coding: utf-8
__version__ = '{major}.{minor}.{patch}'.format(major=0, minor=19, patch=3)
# noinspection PyUnresolvedReferences
from .database import *
# noinspection PyUnresolvedReferences
from .slicer import *
# noinspection PyUnresolvedReferences
from .slicer.widgets import *

__version__ = '1.0.0.dev13'
17 changes: 0 additions & 17 deletions fireant/dashboards/__init__.py

This file was deleted.

76 changes: 0 additions & 76 deletions fireant/dashboards/managers.py

This file was deleted.

117 changes: 0 additions & 117 deletions fireant/dashboards/schemas.py

This file was deleted.

9 changes: 3 additions & 6 deletions fireant/database/__init__.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
# coding: utf-8

from .database import Database
from .base import Database
from .mysql import MySQLDatabase
from .redshift import RedshiftDatabase
from .postgresql import PostgreSQLDatabase
from .vertica import (Vertica,
VerticaDatabase)
from .redshift import RedshiftDatabase
from .vertica import VerticaDatabase
19 changes: 13 additions & 6 deletions fireant/database/database.py → fireant/database/base.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
# coding: utf-8

import pandas as pd

from pypika import Query
from pypika import (
Query,
terms,
)


class Database(object):
"""
WRITEME
"""
# The pypika query class to use for constructing queries
query_cls = Query

Expand All @@ -15,7 +19,7 @@ def connect(self):
def trunc_date(self, field, interval):
raise NotImplementedError

def date_add(self, date_part, interval, field):
def date_add(self, field: terms.Term, date_part: str, interval: int):
""" Database specific function for adding or subtracting dates """
raise NotImplementedError

Expand All @@ -25,6 +29,9 @@ def fetch(self, query):
cursor.execute(query)
return cursor.fetchall()

def fetch_dataframe(self, query):
def fetch_data(self, query):
with self.connect() as connection:
return pd.read_sql(query, connection)
return pd.read_sql(query, connection, coerce_float=True, parse_dates=True)

def totals(self, query, terms):
raise NotImplementedError
24 changes: 12 additions & 12 deletions fireant/database/mysql.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
# coding: utf-8
import pandas as pd

from pypika import (
Dialects,
MySQLQuery,
terms,
)

from fireant.database import Database
from .base import Database


class Trunc(terms.Function):
Expand Down Expand Up @@ -51,25 +50,26 @@ def __init__(self, database=None, host='localhost', port=3306,
def connect(self):
import pymysql

return pymysql.connect(
host=self.host, port=self.port, db=self.database,
user=self.user, password=self.password,
charset=self.charset,
cursorclass=pymysql.cursors.Cursor,
)
return pymysql.connect(host=self.host, port=self.port, db=self.database,
user=self.user, password=self.password,
charset=self.charset,
cursorclass=pymysql.cursors.Cursor)

def fetch(self, query):
with self.connect().cursor() as cursor:
cursor.execute(query)
return cursor.fetchall()

def fetch_dataframe(self, query):
def fetch_data(self, query):
return pd.read_sql(query, self.connect())

def trunc_date(self, field, interval):
return Trunc(field, interval)
return Trunc(field, str(interval))

def date_add(self, date_part, interval, field):
def date_add(self, field, date_part, interval):
# adding an extra 's' as MySQL's interval doesn't work with 'year', 'week' etc, it expects a plural
interval_term = terms.Interval(**{'{}s'.format(str(date_part)): interval, 'dialect': Dialects.MYSQL})
return DateAdd(field, interval_term)

def totals(self, query, terms):
raise NotImplementedError
27 changes: 14 additions & 13 deletions fireant/database/postgresql.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,20 @@
# coding: utf-8
import pandas as pd

from pypika import (
PostgreSQLQuery,
functions as fn,
terms,
)

from fireant.database import Database
from .base import Database


class Trunc(terms.Function):
class DateTrunc(terms.Function):
"""
Wrapper for the PostgreSQL date_trunc function
"""

def __init__(self, field, date_format, alias=None):
super(Trunc, self).__init__('date_trunc', date_format, field, alias=alias)
super(DateTrunc, self).__init__('DATE_TRUNC', date_format, field, alias=alias)
# Setting the fields here means we can access the TRUNC args by name.
self.field = field
self.date_format = date_format
Expand All @@ -26,6 +25,7 @@ class PostgreSQLDatabase(Database):
"""
PostgreSQL client that uses the psycopg module.
"""

# The pypika query class to use for constructing queries
query_cls = PostgreSQLQuery

Expand All @@ -40,21 +40,22 @@ def __init__(self, database=None, host='localhost', port=5432,
def connect(self):
import psycopg2

return psycopg2.connect(
host=self.host, port=self.port, dbname=self.database,
user=self.user, password=self.password,
)
return psycopg2.connect(host=self.host, port=self.port, dbname=self.database,
user=self.user, password=self.password)

def fetch(self, query):
with self.connect().cursor() as cursor:
cursor.execute(query)
return cursor.fetchall()

def fetch_dataframe(self, query):
def fetch_data(self, query):
return pd.read_sql(query, self.connect())

def trunc_date(self, field, interval):
return Trunc(field, interval)
return DateTrunc(field, str(interval))

def date_add(self, field, date_part, interval):
return fn.DateAdd(str(date_part), interval, field)

def date_add(self, date_part, interval, field):
return fn.DateAdd(date_part, interval, field)
def totals(self, query, terms):
raise NotImplementedError
Loading