Webviz on remote bag

[NicolasBernard456]

Hello,

Thanks for your amazing work!
I was wondering if i could get some more infos on how to use webviz on remote bags? I saw that it was possible to do it with a cloud provider like S3 or GCS https://github.com/cruise-automation/webviz/blob/master/packages/webviz-core/src/dataProviders/BrowserHttpReader.js
I have a custom NAS where my bags are stored and I am using Marv as a bag management platform. I wanted to generate link like this to have a complete view of the bags. Can you give me some advices on how to proceed?

If this appears to be too harsh to setup, is there a way to generate an url to have a view of one local bag instead of drag/dropping it?

Best regards,
Nicolas

[janpaul123]

Hi! I don't know much about MARV. Is it able to provide URLs that link to bags, e.g. like https://open-source-webviz-ui.s3.amazonaws.com/demo.bag? If so, then it might just be a matter of configuring their webserver to support CORS and HTTP Range Requests.

I'm trying their demo now, but can't seem to download bag files. E.g. if I click on the bag file link in https://ternaris.com/marv-robotics/demo/#/detail/sdcokcphyel4i52ggeemts33ly then Chrome shows me an "ERR_INVALID_RESPONSE" error. Any idea what might be going on? It would be cool to perhaps make a contribution to MARV to add an "open in Webviz" link.

[janpaul123]

If this appears to be too harsh to setup, is there a way to generate an url to have a view of one local bag instead of drag/dropping it?

Oh I forgot to answer this question. The browser does not have direct access to the file system (for security reasons), so we can't access arbitrary files through a URL. They have to be explicitly selected by the user, either by drag and drop or through a file selection dialog (which is very similar to drag and drop so we didn't bother implement that).

[stefan-enway]

you could try to put https://min.io/ in front of your NAS as it acts as a S3 compatible storage layer.

[NicolasBernard456]

Thank for your answers!
@janpaul123: MARV can provide URLs that link to bag, i've tried it locally it works fine :) I'm not sure why the demo is not working though... I'll try to configure the webserver as you said!

@stefan-enway: Thanks, i'll look into it if I struggle too much setting up the webserver :)

[NicolasBernard456]

I managed to use flask_cors as MARV is implemented in python. I think I got the correct configuration, here is the response header using Postman :

But I still have this error on my console :

I also tried putting http://webviz.io as the origin or localhost:8080 as the origin(and building and running locally) but I'm getting the same result :(

Maybe I am missing something?

[janpaul123]

Looks like CORS headers are not set properly for OPTIONS requests (CORS preflight), try seeing what you get in postman if you make an OPTIONS request.

[stefan-enway]

@NicolasBernard456
docker run -p 9000:9000 -v /mnt/data:/data minio/minio server /data
plus another for nginx with TLS proxy & all the CORS/proxy tuning.
I even automated the letencrypt part for "offical" certs ;)

[NicolasBernard456]

@janpaul123
This is what I get :

[NicolasBernard456]

@stefan-enway Got this error when using min.io

on console :

I tried drag and dropping that bag, it worked fine.. :/

[stefan-enway]

please run curl -H "Access-Control-Request-Method: GET" -H "Origin: http://localhost" --head http://www.google.com replace google with your URI.
and please just copy&paste the output in text (maybe with `` enclosing)

[stefan-enway]

and get an nginx somewhere and look at https://enable-cors.org/server_nginx.html. beware that * is a wildcard for everywhere, but if you use it on a local VM/network you should be fine.

[janpaul123]

@janpaul123
This is what I get

Right so if in that case Access-Control-Allow-Origin and Access-Control-Expose-Headers are missing, then that is the issue!

[NicolasBernard456]

Thanks for your help, I added those headers and it's working now!

[janpaul123]

@NicolasBernard456 any chance you could share your final solution? Would be cool to contribute this back to MARV, too! 😄

[stefan-enway]

oh yeah. in nginx config please ;)

[janpaul123]

@NicolasBernard456 Someone is asking about something similar in #410; do you happen to have a config file that you could share with us here?

[NicolasBernard456]

Hello,
Very sorry for the late reply, I'm using python flask and cross origin. Those are the header that I used:
@cross_origin(origins = 'https://webviz.io', methods = ['GET', 'HEAD', 'OPTIONS'], expose_headers = ['ETag', 'Content-Type', 'Accept-Ranges', 'Content-Length', 'Range'], allow_headers = ['*'], send_wildcard = True, vary_header = True)

[janpaul123]

No worries, and great, thanks for sharing! Btw, I'm trying to get to know our users better, so if you'd be interested in a quick call some time soon feel free to shoot me an email at jp.posma@getcruise.com :)

[bazi88]

I am using nodejs ( express) for this issue and this file is running

`var express = require('express')
var cors = require('cors')
var app = express()

crossOption = {
"origin" : '',
"methods" : ['GET', 'HEAD', 'OPTIONS'],
"exposedHeaders" : ['ETag', 'Content-Type', 'Accept-Ranges', 'Content-Length', 'Range'],
"allowedHeaders" : [''],
}
app.use(cors(crossOption))

//Using direct file URL
app.get('/demo.bag',function(req,res){
res.download(__dirname +'/2021-01-11-18-26-43.bag');
})
app.listen(9999, function () {
console.log('CORS-enabled web server listening on port 80')
})`

It's working.

[hefc2016]

@NicolasBernard456
Hi,

I also want to use marv as data managament platfrom for bag files and visulaize with webviz via remote url.

But i'm new with web server and don't know how to create URLs that link to bag, and config CORS with latest marv which has switched to use synchronous HTTP client/server framework aiohttp.

Could you share your marv code to me that I can try and study?

Many thanks.

[lanyusea]

leave a comment for people who is working on django:

I'm trying to use django and django-cors-headers to setup a file-hosting server for the remote bag accessing. The setup looks good and tested by curl request. However, in the developer console of browser, header Access-Control-Allow-Origin is still missing.

After some searching, this answer mentions that

Django does not serve the media files through the middleware stack, so any attempts to inject headers are pointless. 

so it means we cannot set up the https://xxx/*.bag by django from which webviz loads remote bags.

And at last I setup a apache server instead, with these settings in .conf file

    Header always set Access-Control-Allow-Methods "GET, HEAD"
    Header always set Access-Control-Allow-Origin "*"
    Header always set Access-Control-Allow-Headers "*"
    Header always set Access-Control-Expose-Headers "ETag, Content-Type, Content-Length, Accept-Ranges"

[C1oudust]

I am using nodejs ( express) for this issue and this file is running

`var express = require('express') var cors = require('cors') var app = express()

crossOption = { "origin" : '', "methods" : ['GET', 'HEAD', 'OPTIONS'], "exposedHeaders" : ['ETag', 'Content-Type', 'Accept-Ranges', 'Content-Length', 'Range'], "allowedHeaders" : [''], } app.use(cors(crossOption))

//Using direct file URL app.get('/demo.bag',function(req,res){ res.download(__dirname +'/2021-01-11-18-26-43.bag'); }) app.listen(9999, function () { console.log('CORS-enabled web server listening on port 80') })`

It's working.

Hi,
I used your code and it works, but when my bag size is very large, webviz will report an error : Requested more data than cache size: 265699599 > 209715200
I wonder if there is a solution?

[ben-z]

I ran into this (self-hosting bags and having CORS issues) today and wanted a 1-line solution. In case anyone else is in the same boat, here's my approach for serving the bag files:

npx 'https://github.com/ben-z/http-server#8aa6a8f' --cors='ETag,Content-Type,Accept-Ranges,Content-Length'

Had to patch the http-server package to add support for the Access-Control-Expose-Headers header.

[jtbandes]

We've also written up some documentation on setting up CORS correctly: https://foxglove.dev/docs/studio/connection/setting-up-cors