Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pipeline #23

Merged
merged 10 commits into from
Mar 9, 2020
Merged

Pipeline #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
eff4d71
feat: init a pipeline structure
ObservedObserver Mar 4, 2020
a94b448
feat: a work demo
ObservedObserver Mar 4, 2020
f0dd5cb
feat: wrap injection params
ObservedObserver Mar 5, 2020
c2eab9c
feat: new intention insight demo
ObservedObserver Mar 5, 2020
ac3b827
feat: outier good demo
ObservedObserver Mar 5, 2020
0f8ba7b
feat: adjust dev vis board
ObservedObserver Mar 6, 2020
3ab7ee8
feat: group intend and dev panel ui adjust
ObservedObserver Mar 6, 2020
3a06921
doc: update current pipeline
ObservedObserver Mar 9, 2020
1f76270
Update README.md
ObservedObserver Mar 9, 2020
51809e9
fix: add hint for no cooked data cases.
ObservedObserver Mar 9, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
16 changes: 10 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ npm i visual-insights --save`
## How does it work
The working process are visualized in notebook board in the application. *** Main process of the algorithm is shown in the `notebook` board. *** Here shows how rath use visual-insights to make a analytic pipeline.

![](https://cdn.nlark.com/yuque/0/2019/png/171008/1570692438037-b2ce208d-bd1d-4b38-be27-9251bbb171d2.png)
![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/rath-arc.png)

### Univariate summary
For the first step, rath analyze all the fields in the dataset independently. It gets the fields' distributions and calculate its entropy. Besides, it will define a semantic type (`quantitative`, `ordinal`, `temporal`, `nominal`) for each field. More details of the field will be displayed when hover your mouse on the fields.
Expand All @@ -96,22 +96,26 @@ For the first step, rath analyze all the fields in the dataset independently. It
Then, it will find the fields with high entropy and try to reduce it by grouping the field (for example). Only dimensions participates this process.

### Subspaces
In this step, visual insights search the combination of fields, and calculate the entropy of each measure with some aggregation operators.
In this step, visual insights search the combination of fields. Visual-Insights suppose that any two fields appears in a view should be correlated with each other otherwise they should be display in seperated view. Visual-Insight now use crammver'V and pearson' cc for different types of fields' correlation.

![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/subspaces.svg)

### Correlation
After one subspace is specified (try to click one row of subsapce in notebook), it will analyze the correlation of measures in the space.
#### Correlation

for example, the correlation of measures:

![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/correlation.svg)

### Clustering
#### Clustering
It helps you to cluster all the measures based on their correlation. It puts all the variables who are strongly related together to make a specific view (with specified dimenions).

Click one group of measures and visualization will be shown at the bottom of the page.

![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/clustering.svg)

### Insight Extraction
After we get many subspaces, we can check the insight significance of each space. Currently, visual-insights support trend, outlier, group(whether different groups of data behave differently for spefic measures)

![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/rath-demo.jpg)

### Specification & Visualization

Expand Down
19 changes: 11 additions & 8 deletions README.zh-CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,31 +90,34 @@ Rath的主要分析流程在dashboard模块中有每个环节的可视化，可
![](https://cdn.nlark.com/yuque/0/2019/png/171008/1570692438037-b2ce208d-bd1d-4b38-be27-9251bbb171d2.png)

### Univariate summary

好累，翻译不动了，改日再来。欢迎pr...
For the first step, visual-insights analyze all the fields in the dataset independently. It gets the fields' distributions and calculate its entropy. Besides, it will define a semantic type (`quantitative`, `ordinal`, `temporal`, `nominal`) for each field. More details of the field will be displayed when hover your mouse on the fields.
For the first step, rath analyze all the fields in the dataset independently. It gets the fields' distributions and calculate its entropy. Besides, it will define a semantic type (`quantitative`, `ordinal`, `temporal`, `nominal`) for each field. More details of the field will be displayed when hover your mouse on the fields.

![](https://cdn.nlark.com/yuque/0/2019/jpeg/171008/1570614609678-33d5f2c1-e51e-4bcd-8343-271a041f7519.jpeg)

Then, it will find the fields with high entropy and try to reduce it by grouping the field (for example). Only dimensions participates this process.

### Subspaces
In this step, visual insights search the combination of fields, and calculate the entropy of each measure with some aggregation operators.
In this step, visual insights search the combination of fields. Visual-Insights suppose that any two fields appears in a view should be correlated with each other otherwise they should be display in seperated view. Visual-Insight now use crammver'V and pearson' cc for different types of fields' correlation.

![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/subspaces.svg)

### Correlation
After one subspace is specified (try to click one row of subsapce in notebook), it will analyze the correlation of measures in the space.
#### Correlation

for example, the correlation of measures:

![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/correlation.svg)

### Clustering
#### Clustering
It helps you to cluster all the measures based on their correlation. It puts all the variables who are strongly related together to make a specific view (with specified dimenions).

Click one group of measures and visualization will be shown at the bottom of the page.

![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/clustering.svg)

### Insight Extraction
After we get many subspaces, we can check the insight significance of each space. Currently, visual-insights support trend, outlier, group(whether different groups of data behave differently for spefic measures)

![](https://chspace.oss-cn-hongkong.aliyuncs.com/visual-insights/rath-demo.jpg)


### Specification & Visualization

Expand Down
7 changes: 6 additions & 1 deletion packages/frontend/src/App.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,15 @@ import Gallery from "./pages/gallery/index";
import NoteBook from "./pages/notebook/index";
import DataSourceBoard from "./pages/dataSource/index";
import DashBoardPage from './pages/dashBoard/index';
import DevPage from './pages/dev';
import UserSettings from './components/userSettings';

const pivotList = [
'DataSource',
'NoteBook',
'Explore',
'DashBoard'
'DashBoard',
'Dev'
].map((page, index) => {
return { title: page, itemKey: 'pivot-' + (index + 1)}
});
Expand Down Expand Up @@ -121,6 +123,9 @@ function App() {
{
pageStatus.current.pivotKey === 'pivot-4' && <DashBoardPage />
}
{
pageStatus.current.pivotKey === 'pivot-5' && <DevPage />
}
</div>
);
}
Expand Down
234 changes: 234 additions & 0 deletions packages/frontend/src/components/radarChart.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,234 @@
import React, { useRef, useEffect, useMemo } from 'react';
import embed from 'vega-embed';
import { scheme } from 'vega';

scheme('threshold', ['#1890ff', '#ffccc7']);

interface RadarChartProps {
keyField: string;
valueField: string;
dataSource: any[];
threshold: number;
}
interface DataRecord {
key: string;
value: number;
category: 0 | 1;
}
const RadarChart: React.FC<RadarChartProps> = props => {
const { keyField, valueField, dataSource = [], threshold } = props;
const container = useRef<HTMLDivElement>(null);
const viewData = useMemo(() => {
let data: DataRecord[] = dataSource.map(record => {
return {
key: record[keyField],
value: Math.round(record[valueField] * 1000) / 1000,
category: 0
}
});
['outlier', 'trend', 'general', 'group'].forEach(type => {
if (!data.find(d => d.key === type)) {
data.push({
key: type,
value: 0,
category: 0
})
}
})
let ruleData: DataRecord[] = data.map(record => {
return {
key: record.key,
value: threshold,
category: 1
}
})
return data.concat(ruleData);
}, [keyField, valueField, dataSource, threshold])
useEffect(() => {
if (container.current) {
embed(container.current, {
width: 280,
height: 280,
padding: 50,
autosize: { type: "none", contains: "padding" },

signals: [{ name: "radius", update: "width / 2" }],

data: [
{
name: "table",
values: viewData
},
{
name: "keys",
source: "table",
transform: [
{
type: "aggregate",
groupby: ["key"]
}
]
}
],

scales: [
{
name: "angular",
type: "point",
range: { signal: "[-PI, PI]" },
padding: 0.5,
domain: { data: "table", field: "key" }
},
{
name: "radial",
type: "linear",
range: { signal: "[0, radius]" },
zero: true,
nice: false,
domain: [0, 1],
domainMin: 0
},
{
name: "color",
type: "ordinal",
domain: { data: "table", field: "category" },
range: { scheme: "threshold" }
}
],

encode: {
enter: {
x: { signal: "radius" },
y: { signal: "radius" }
}
},

marks: [
{
type: "group",
name: "categories",
zindex: 1,
from: {
facet: { data: "table", name: "facet", groupby: ["category"] }
},
marks: [
{
type: "line",
name: "category-line",
from: { data: "facet" },
encode: {
enter: {
interpolate: { value: "linear-closed" },
x: {
signal:
"scale('radial', datum.value) * cos(scale('angular', datum.key))"
},
y: {
signal:
"scale('radial', datum.value) * sin(scale('angular', datum.key))"
},
stroke: { scale: "color", field: "category" },
strokeWidth: { value: 1 },
fill: { scale: "color", field: "category" },
fillOpacity: { value: 0.1 }
}
}
},
{
type: "text",
name: "value-text",
from: { data: "category-line" },
encode: {
enter: {
x: { signal: "datum.x" },
y: { signal: "datum.y" },
text: { signal: "datum.datum.value" },
align: { value: "center" },
baseline: { value: "middle" },
fill: { value: "#262626" }
}
}
}
]
},
{
type: "rule",
name: "radial-grid",
from: { data: "keys" },
zindex: 0,
encode: {
enter: {
x: { value: 0 },
y: { value: 0 },
x2: { signal: "radius * cos(scale('angular', datum.key))" },
y2: { signal: "radius * sin(scale('angular', datum.key))" },
stroke: { value: "lightgray" },
strokeWidth: { value: 1 }
}
}
},
{
type: "text",
name: "key-label",
from: { data: "keys" },
zindex: 1,
encode: {
enter: {
x: {
signal: "(radius + 5) * cos(scale('angular', datum.key))"
},
y: {
signal: "(radius + 5) * sin(scale('angular', datum.key))"
},
text: { field: "key" },
align: [
{
test: "abs(scale('angular', datum.key)) > PI / 2",
value: "right"
},
{
value: "left"
}
],
baseline: [
{
test: "scale('angular', datum.key) > 0",
value: "top"
},
{
test: "scale('angular', datum.key) == 0",
value: "middle"
},
{
value: "bottom"
}
],
fill: { value: "black" },
fontWeight: { value: "bold" }
}
}
},
{
type: "line",
name: "outer-line",
from: { data: "radial-grid" },
encode: {
enter: {
interpolate: { value: "linear-closed" },
x: { field: "x2" },
y: { field: "y2" },
stroke: { value: "lightgray" },
strokeWidth: { value: 1 }
}
}
}
]
} as any, {
actions: false
});
}
}, [viewData]);
return <div ref={container}></div>
}

export default RadarChart;