Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(ResNet) Support CrossEntropyCriterion Layer in Spark-DL #7

Closed
yiheng opened this issue Sep 14, 2016 · 0 comments
Closed

(ResNet) Support CrossEntropyCriterion Layer in Spark-DL #7

yiheng opened this issue Sep 14, 2016 · 0 comments

Comments

@yiheng
Copy link
Contributor

yiheng commented Sep 14, 2016 •
edited

We need to support CrossEntropyCriterion layer to enable resnet. We can reference the torch implementation at https://github.com/torch/nn/blob/master/CrossEntropyCriterion.lua.
@yiheng yiheng changed the title Support CrossEntropyCriterion Layer in Spark-DL (ResNet) Support CrossEntropyCriterion Layer in Spark-DL Sep 14, 2016
@yiheng yiheng closed this as completed Sep 29, 2016
i8run referenced this issue in i8run/BigDL Feb 9, 2018
@zhangxiaoli73
Merge pull request #7 from i8run/open-source-mkl-dnn-for-cherry
d27c338 
memory leak and update bn performance
i8run added a commit that referenced this issue Jun 28, 2018
@i8run
Feat: MKL-DNN Supports (#2482)
5f18519 
This feature enables mkl-dnn support, which can speed up deep learning model. We wrapper the native c api in the java, which are in BigDL-core projects. And in BigDL, we integrated the convolution, batchnorm, maxpooling, avgpooling, relu, lrn, softmax, caddtable and concattable. Currently, it  supports create the model which only contains dnn layer or container.

Because the data layout is optimized in mkl-dnn. The mkl-dnn model will use `DnnTensor` which contains the native buffer as a default tensor. So there're some notations,

1. User should copy the data from jvm heap at the first layer and copy back to jvm heap at the last layer.
2. User should compile the model, which contains the phase (training/inference) and input tensor size. It will infer and allocate the other information.

* fix: linear performance issue and serialization of java object in MklDnnTensor

* memory leak refactor

* memory leak and bn performance issues

1. Memory Leak
The internal buffer with MklDnnTensor should not be re-assigned without
releasing. So we should check it first. At first iteration or after the
changing of input size, we create a new MklDnnTensor as a buffer.

2. Bn perf
The JIT BatchNormalization only supports avx2 or avx512, which has much
batter performance than ref version. The input and gradOutput format
should be the same to get the best performance.

* test: add some test cases for BatchNorm.

The computation of float value is not the same as C/C++/Native with JVM.
And batch norm will make it much greater such as 10^-8 -> 10^-4 -> 10^-1

* fix: rebase with upstream master:

1. Concat and ConcatTable should inherit from DynamicContainer.
2. updateParameters has been depricated.
3. zeroGradParameters should be final. But from now on, the Linear
   should use it.
4. Some other syntax or semantic errors.

* perf: single node and single model performance

* perf: single model

* feat: add fusion for mkl-dnn

* test: add test utils to compare dnn output

* test: add some tests compared with caffe

* add unit tests for dnn tensor

* add unit test for reorder memory

* test: fix the test regression errors

* checkin reorder manager

* add backward for sequential

* fix some bugs

* update core ref

* add unit tests

* refactor: move the static class DataType, AlgKind and so on to standalone class (#4)

* refactor: delete MklDnn.MemoryFormat

* refactor: move the static class DataType, AlgKind and so on to standalone class

* fix: core refactor errors

* refactor: spec errors (#5)

* Mkl dnn dev (#6)

* checkin reorder manager

* add container and refine reorder manager

* fix merge issue

* add join table forward

* refine inteface (#7)

* add LRN and ReLU

* add pooling

* refactor: conv + linear + bn

* add JoinTable backward

* refactor: conv + linear + bn

* add cAddTable concattable

* fix: reorder failed on some of convs

* refactor: softmax

* refactor: fusion support

* refactor: resnet_50

* refactor: move tests to this branch

* refactor: delete unusefull files and enable the special old tests.
refactor: delete unsed methods in MklDnnOps
fix: scalastyle check

* fix: rebase with upstream

* fix: ignore the prototxt tests

* fix: do not change the core commit ref

* fix: move set num of threads for mkldnn to ResNet50Perf

* fix: serialization disabled for mkldnn module
wzhongyuan pushed a commit to wzhongyuan/BigDL that referenced this issue Jan 16, 2019
@i8run
fix: make mkl with sequential mode which has much better performance (i…
a51b47f 
…ntel-analytics#7)
wzhongyuan pushed a commit to wzhongyuan/BigDL that referenced this issue Jan 16, 2019
@i8run
[WIP] MKL-DNN directly wrapper for BigDL. (intel-analytics#55)
1bc0908 
* feat: mkl-dnn initialize

* fix: structure of building

* fix: public final static

* fix: delete the dependencies of environments

* fix: skip tests

* add update dnn wrappers

* fix: dynamic load iomp5

* feat linear supports and some fix

* add more wrapper

* add lrn api

* fix: add bn and softmax

* fix: some fixes

* fix: mkl-dnn build

* feat: add get format api

* fix: add getSize

* feat: aligned memory

* add conv fuse relu api

* fix: add aligned storage

* add concat api

* fix: mkl envs for lib mkldnn

* fix: add mkl add method with 2 ptrs

* fix: update to Release

* fix: batch norm infer mode

* fix: update 0.5.0 -> 0.6.0

* add free (intel-analytics#5)

* feat: affinity for java thread

* fix: update core branch

* fix: delete the memset constant value for debug, and add affinity

* feat: add mkl-dnn fusion

* fix: memory format enum consistent with dnn

* feat: add auto format

* refactor: delete the MemoryFormat in MklDnn

* Memory should load MKLDnn (intel-analytics#6)

* refactor: move enums to seprate classes (intel-analytics#7)

* feat: add GetShape and GetFormat api

* fix: delete printf

* fix a bug

* add sum

* refactor: change name

* refactor: change submodule infos

* fix: set block time by default. A property to control to disable it
AndyLuo1029 pushed a commit to AndyLuo1029/BigDL that referenced this issue Aug 30, 2022
@shane-huang
Merge pull request intel-analytics#7 from AndyLuo1029/github-landing
ee00896 
GitHub landing
Litchilitchy added a commit to Litchilitchy/BigDL that referenced this issue Sep 23, 2022
@Litchilitchy
Merge ckks in FLServer LR Aggregator (intel-analytics#7)
a948a81 
* update ckks in FLServer LR Aggregator

* some changes

* support ckks at scala FL Server

* add

* add

* fix and add ut

* fix and add ut

* add support for customized client module and divide example to 2 parties

* add comments
qiuxin2012 added a commit that referenced this issue Oct 13, 2022
@qiuxin2012 @Litchilitchy
vfl LR ckks (#6001)
b388e75 
* vfl LR ckks (#10)

* tests

* add census

* add train

* update census

* update

* add time count

* add new layer

* update vfl client

* add decrypt

* ckksTest

* Merge ckks in FLServer LR Aggregator (#7)

* update ckks in FLServer LR Aggregator

* some changes

* support ckks at scala FL Server

* add

* add

* fix and add ut

* fix and add ut

* add support for customized client module and divide example to 2 parties

* add comments

* split

* Ckks & Dllib Example fix (#8)

* update ckks in FLServer LR Aggregator

* some changes

* support ckks at scala FL Server

* add

* add

* fix and add ut

* fix and add ut

* add support for customized client module and divide example to 2 parties

* add comments

* fix

* fix none target sample

* fix model

* fix train

* fix accuracy

* add shuffle

* enable ckks

* fix ckks

* Ckks fix the bug and add server error message (#9)

* update ckks in FLServer LR Aggregator

* some changes

* support ckks at scala FL Server

* add

* add

* fix and add ut

* fix and add ut

* add support for customized client module and divide example to 2 parties

* add comments

* fix

* fix

* some fix

* update

Co-authored-by: Song Jiaming <litchy233@gmail.com>

* remove useless  dependency

* update client

* update client

* update

* fix style

* update

* update unit test

* code clean up

* fix ut

* add all-dependencies

* update

* update readme

* update readme and pom

* fix ut

* update pom

* update pom

* update

* add documents

* some update

* revert some code

Co-authored-by: Song Jiaming <litchy233@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yiheng @JerryYanWan