Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing nft #1

Open
wants to merge 89 commits into
base: nft
from

Conversation

@kalwalt
Copy link
Owner

commented May 28, 2019

My goal is to make a stable NFT branch;

  • updating to the latest artoolkit5
  • upgrading all the methods
  • improving and finding the bottlneck in the code
  • restoring the wasm example
  • fixing the FS not defined issue
  • upgrading the Three.js version to r103
  • restoring the webWorker version

For anyone interested to try and to build the lib:
The jsartoolkit is based on the emscripten compiler, you need to install it in your machine, follow these instructions: https://emscripten.org/docs/getting_started/downloads.html. i used emscripten 1.38.31 .
If you want to build jsartoolkit5 use this version of libjpeg:
https://github.com/kalwalt/libjpeg-for-jsartoolkit5

kalwalt added 20 commits Feb 6, 2019
@kalwalt kalwalt added the enhancement label May 28, 2019
@kalwalt kalwalt self-assigned this May 28, 2019
@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented May 28, 2019

with this last commit it compile but i receive this error, in nft_inti_test.html example:

Navigated to http://localhost:3000/examples/nft_init_test.html
nft_init_test.html:53 Live reload enabled.
artoolkit.api.js:1647 Uncaught TypeError: Module._loadCamera is not a function
    at writeCallback (artoolkit.api.js:1647)
    at writeByteArrayToFS (artoolkit.api.js:1674)
    at XMLHttpRequest.oReq.onload (artoolkit.api.js:1690)
writeCallback @ artoolkit.api.js:1647
writeByteArrayToFS @ artoolkit.api.js:1674
oReq.onload @ artoolkit.api.js:1690
load (async)
ajax @ artoolkit.api.js:1686
loadCamera @ artoolkit.api.js:1655
ARCameraParam.load @ artoolkit.api.js:1429
(anonymous) @ nft_init_test.html:22

in nft_threejs.html:

Navigated to http://localhost:3000/examples/nft_threejs.html
nft_threejs.html:149 Live reload enabled.
artoolkitNft.min.js:1 missing function: arVideoGetPixelFormat
_arVideoGetPixelFormat @ artoolkitNft.min.js:1
_q @ artoolkitNft.min.js:11
Rq @ artoolkitNft.min.js:11
Qq @ artoolkitNft.min.js:11
Ep @ artoolkitNft.min.js:11
ws @ artoolkitNft.min.js:11
eK @ artoolkitNft.min.js:12
dynCall_iiiii_7 @ VM619:4
setup @ VM664:10
ARController._initialize @ artoolkitNft.min.js:1
ARController @ artoolkitNft.min.js:1
(anonymous) @ artoolkitNft.min.js:1
(anonymous) @ artoolkitNft.min.js:1
writeCallback @ artoolkitNft.min.js:1
writeByteArrayToFS @ artoolkitNft.min.js:1
oReq.onload @ artoolkitNft.min.js:1
load (async)
ajax @ artoolkitNft.min.js:1
loadCamera @ artoolkitNft.min.js:1
ARCameraParam.load @ artoolkitNft.min.js:1
ARCameraParam @ artoolkitNft.min.js:1
obj.onSuccess @ artoolkitNft.min.js:1
initProgress @ artoolkitNft.min.js:1
artoolkitNft.min.js:18 -1
artoolkitNft.min.js:18 -1
abort @ artoolkitNft.min.js:18
_arVideoGetPixelFormat @ artoolkitNft.min.js:1
_q @ artoolkitNft.min.js:11
Rq @ artoolkitNft.min.js:11
Qq @ artoolkitNft.min.js:11
Ep @ artoolkitNft.min.js:11
ws @ artoolkitNft.min.js:11
eK @ artoolkitNft.min.js:12
dynCall_iiiii_7 @ VM619:4
setup @ VM664:10
ARController._initialize @ artoolkitNft.min.js:1
ARController @ artoolkitNft.min.js:1
(anonymous) @ artoolkitNft.min.js:1
(anonymous) @ artoolkitNft.min.js:1
writeCallback @ artoolkitNft.min.js:1
writeByteArrayToFS @ artoolkitNft.min.js:1
oReq.onload @ artoolkitNft.min.js:1
load (async)
ajax @ artoolkitNft.min.js:1
loadCamera @ artoolkitNft.min.js:1
ARCameraParam.load @ artoolkitNft.min.js:1
ARCameraParam @ artoolkitNft.min.js:1
obj.onSuccess @ artoolkitNft.min.js:1
initProgress @ artoolkitNft.min.js:1
artoolkitNft.min.js:18 Uncaught abort(-1). Build with -s ASSERTIONS=1 for more info.
abort @ artoolkitNft.min.js:18
_arVideoGetPixelFormat @ artoolkitNft.min.js:1
_q @ artoolkitNft.min.js:11
Rq @ artoolkitNft.min.js:11
Qq @ artoolkitNft.min.js:11
Ep @ artoolkitNft.min.js:11
ws @ artoolkitNft.min.js:11
eK @ artoolkitNft.min.js:12
dynCall_iiiii_7 @ VM619:4
setup @ VM664:10
ARController._initialize @ artoolkitNft.min.js:1
ARController @ artoolkitNft.min.js:1
(anonymous) @ artoolkitNft.min.js:1
(anonymous) @ artoolkitNft.min.js:1
writeCallback @ artoolkitNft.min.js:1
writeByteArrayToFS @ artoolkitNft.min.js:1
oReq.onload @ artoolkitNft.min.js:1
load (async)
ajax @ artoolkitNft.min.js:1
loadCamera @ artoolkitNft.min.js:1
ARCameraParam.load @ artoolkitNft.min.js:1
ARCameraParam @ artoolkitNft.min.js:1
obj.onSuccess @ artoolkitNft.min.js:1
initProgress @ artoolkitNft.min.js:1
@@ -48,7 +50,7 @@ struct arController {
int patt_id = 0; // Running pattern marker id

ARdouble cameraLens[16];

AR_PIXEL_FORMAT pixFormat = arVideoGetPixelFormat();

This comment has been minimized.

Copy link
@kalwalt

kalwalt May 28, 2019

Author Owner

see this comment #1 (comment)

@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Jun 2, 2019

With the latest, the examples runs but patterns are not recognized. No errors from the console.

@kalwalt kalwalt referenced this pull request Jul 22, 2019
3 of 8 tasks complete
@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Jul 22, 2019

@nicolocarpignoli @evaristoc @ThorstenBux @shawmakesmusic I'm adding threading support for NFT in this PR #2 there are good possibility for a real improvements!

@nicolocarpignoli

This comment has been minimized.

Copy link

commented Jul 22, 2019

great job! looking forward to it

@Carnaux

This comment has been minimized.

Copy link

commented Jul 24, 2019

I'm trying to port the genTexData.c, using wasm, and create a online ntf marker creator, the git repo is here. I'll update it as i make progress, it does not looks complicated

@nicolocarpignoli

This comment has been minimized.

Copy link

commented Jul 24, 2019

@Carnaux it will be great! With enhanced performance and a ready-to-use nft marker creator this could really be a thing!

@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Jul 24, 2019

@Carnaux That's a food idea! Ping me in your repo or PR if you Need a help. Are you working to Port It with Emscripten?
And also post in the Ar.js issue tracker for more visibility.

@nicolocarpignoli

This comment has been minimized.

Copy link

commented Jul 25, 2019

I will report this on AR.js issue, maybe someone can help if needed

@Carnaux

This comment has been minimized.

Copy link

commented Jul 25, 2019

@nicolocarpignoli Nice, thanks, @kalwalt yes with emscripten, i'm generating the .iset file with any image from a input tag, so the fset and the fset3 will be generated with no problem, now I'm trying to create the files and download them.

@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Jul 25, 2019

@nicolocarpignoli Nice, thanks, @kalwalt yes with emscripten, i'm generating the .iset file with any image from a input tag, so the fset and the fset3 will be generated with no problem, now I'm trying to create the files and download them.

Well done @Carnaux , i will follow your results 😄

@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Jul 29, 2019

@nicolocarpignoli @evaristoc @shawmakesmusic @Carnaux @ThorstenBux I restored the webworker version, test please the nft_worker_threejs.html example. I made a short test only on my Desktop PC, i got 50 fps and a bit less while detecting the pinball image. The problem is that the sphere object moves jerkily with jagging steps, I'm wondering how we can fix and improve it.
Compared to the 'threaded' version in my nft-with-threads branch the fps are almost the same (on Desktop pc)...

@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Jul 30, 2019

The performaces are not better yet but i'm working to extend the webWorkers, adding some crucial part of the matching process in a new worker. I never did something with webWorkers, so i'm learning while coding, do not expect great things, If you want to test, test the nft_worker_threejs.html example.

@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Jul 30, 2019

With e613291
i added a not threaded version of ar2Tracking() function. Not sure if it is a good or a bad idea: For me it's a starting point for future comparisons. Maybe i will delete it or i will modify it.

kalwalt
@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Aug 8, 2019

The last commit to change the libjpeg, because my hard disk went done and i lost all my data, reinstalling from a backup can't find my initial lib for libjpeg but after some trial and error, and searching i solved it.
Anyway the name jpeg-6b was confusing i think has nothing to do ( maybe this version was used in the past?) If you want to build by yourself use this libjpeg for jsartoolkit5:
https://github.com/kalwalt/libjpeg-for-jsartoolkit5
i will update also the header infos for this PR.
Also the commit is unverified, but it's me i changed the gpg key and it is not recognized? don't know why this happens...

kalwalt added 2 commits Aug 14, 2019
@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Aug 30, 2019

I think if we can find some sort of optimization is in weborker + wasm. I also have to find why the device hang for a long time while initing, it hurts a lot!
I deleted the trackingMod file because it slow down too much, it has no sense to keep that part of the code.

@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Aug 30, 2019

i made this performance test on my pc, with the nft_threejs.html example:


  • pinball.fset 325ms 12.3KB
  • pinball.iset 99ms 125.9KB
  • pinball.fset3 2.81s 640.3KB

the pinball.fset3 file is the longest to load, maybe this is causing to hang in the mobile?

@ThorstenBux

This comment has been minimized.

Copy link

commented Aug 30, 2019

@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Aug 30, 2019

@ThorstenBux yes only loading time not tracking, just wondering why so much time. On mobile it takes much longer. Too much, infact it hangs for much than 6-8 seconds. I will make the same test on Smartphone devices too.

@evaristoc

This comment has been minimized.

Copy link

commented Sep 3, 2019

@kalwalt @ThorstenBux

The training pattern that is left in the repository, I think it is already trained, isn't it?

The procedure will read the descriptors that are saved in the data files, then will probably transform and check every frame of the target video to find similarities.

The more the descriptors, the more time to make the preliminary calculations.

I would suggest to check what it is actually affecting the delay in the loading: either the treatment of the descriptors of the training pattern, or the reading, transformation and subsequent comparison of the frames.

It is likely that every frame will be evaluated as new: catching is difficult, specially because it is about a moving object. However, it is possible to implement some kind of frame comparison before detecting the points. If the target frame doesn't change respect to the previous one, then avoid re-evaluating the image (=> keep the 3D model on the same position).

I haven't tried this one for my project. Hmmm... I think I will...

@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Sep 11, 2019

@evaristoc Sorry for the delay to answer here, but i was on holydays for few days.
Regarding the first question: they should be pre-trained patterns. As i said, i tested on my Desktop Linux machine, i would do on a Android mobile because it seems It tooks a lot more, 9-10 seconda too much in my opinion; the loading of the trainer data should be async but seems blocking the app. This needs to be investigated... I hope to do this in this next days ☺️

@evaristoc

This comment has been minimized.

Copy link

commented Sep 13, 2019

@kalwalt hope you had nice holidays!

Ok... if the code is doing some calculations while loading, then that might block the loop? It is Javascript after all. Isn't it?

Also, assuming it is the loading trained pattern file I wonder if a more suitable data structure can be helpful to find efficiencies. Here some ideas:

  • If there are transformations that always repeat themselves, they could be done at pre-training and somehow be in that file.
  • Can we think on a selective/hierarchical approach for loading? Eg. we could see if there are:
    • characteristic records that will never be used for matching, or
    • characteristic records that might have less value for matching, or
    • characteristic records that require more/less calculation with the code in use

It must be universally applicable though: all files treated the same should always provide a better efficiency for your case.

@kalwalt

This comment has been minimized.

Copy link
Owner Author

commented Sep 14, 2019

@kalwalt hope you had nice holidays!
thank you @evaristoc was beautiful days at the sea!

I don't know if the code is doing some calculation, but it could be. Regarding to the code: the ajax function is almost pure javascript and emscripten code.
About the NFT data structure: i will investigate, i have no a clear idea on how the data is treated. For now I thank you for the input.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.