Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to vc.widget() in my server #88

Closed
FADHLyemen opened this issue Jul 24, 2021 · 37 comments
Closed

how to vc.widget() in my server #88

FADHLyemen opened this issue Jul 24, 2021 · 37 comments

Comments

@FADHLyemen
Copy link

Hi,
I follow steps in:
https://github.com/vitessce/vitessce-python/blob/master/docs/notebooks/widget_brain.ipynb
when I run on university hpc server using

vw=vc.widget()
vw

I got Running on http://127.0.0.1:8002
and error in chrome:
This site can’t be reached127.0.0.1 refused to connect.

Any suggestion how to access the vw?

Thank you

BTW:
I created the following conda env

$ git clone https://github.com/vitessce/vitessce-python.git
cd docs/notebooks
conda env create -f environment.yml
conda activate vitessce-jupyter-examples
pip install -e ../..
jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install ../../js

then
jupyter lab

@keller-mark
Copy link
Member

Hi Fadhl,
If you are running the Jupyter server on the HPC and then accessing the widget through visiting the Jupyter client on your personal computer, can you try the following:

  1. install the jupyter-server-proxy package
  2. set the proxy=True flag when you run the widget method
    vw=vc.widget(proxy=True)
    vw
    

@FADHLyemen
Copy link
Author

Thank you, @keller-mark
I tried it but does not work.

image

image

image

I also added 8002 as a tunnel port in puTTy. but still get the same.

I think there is something wrong. Could you help?

@FADHLyemen
Copy link
Author

is it because:
Error displaying widget model not found

@keller-mark
Copy link
Member

I apologize, I think our documentation is missing a step for bundling the JavaScript file for the widget. Can you try:

cd js
npm install
npm run build
cd ../docs/notebooks
jupyter labextension install ../../js

@FADHLyemen
Copy link
Author

thank you,
npm instatll gave me many warrnings
image

npm run build
image

BTW I installed nodeJs by
conda install -c conda-forge nodejs

@FADHLyemen
Copy link
Author

@keller-mark any help is appreciated

@ilan-gold
Copy link
Collaborator

@FADHLyemen What version of node and/or npm do you have? Could you provide that debug log file (or a more complete error) - hard to tell what's going on since the screenshot is truncated. Thanks!

@ilan-gold
Copy link
Collaborator

@FADHLyemen I see you only have node 6. @keller-mark can correct me, but I believe you probably need node 14

@ilan-gold
Copy link
Collaborator

See https://github.com/vitessce/vitessce-python/blob/master/.github/workflows/test.yml for how we set up an example environment in our CI workflow

@keller-mark
Copy link
Member

Thank you @ilan-gold. Yes, I would also check that you have NodeJS version 14. We should note this in the development section of the README, or point users to the JS repo's README for that step.

Also, I just made this pull request #89 which allows using the vitessce.io web app in place of the Jupyter widget, in case you are interested. That would avoid the need to build the JS files locally, (but opens Vitessce in a separate tab rather than embedded in the notebook).

@FADHLyemen
Copy link
Author

$ node -v
v13.13.0

BTW:
where is web_app_brain.ipynb to try?

sorry for this, I saw one of your team member presented at ISMB this tool:
http://avivator.gehlenborglab.org/?image_url=https://viv-demo.storage.googleapis.com/LuCa-7color_Scan1

can you do the same for Viteesce, so we only need to upload our h5ad file?

@ilan-gold
Copy link
Collaborator

ilan-gold commented Jul 27, 2021

@FADHLyemen For Vitessce, we only accept zarr backed AnnData so an h5ad file would need to be converted via write_zarr. We have considered building wrapper tools around AnnData stores (similar to what Avivator does for OME) to make access easier than having to set up a view config, but for now it hasn't been something we've focused on.

@ilan-gold
Copy link
Collaborator

ilan-gold commented Jul 27, 2021

where is web_app_brain.ipynb to try?

That is in a PR currently #89 and will be merged eventually

@ilan-gold
Copy link
Collaborator

As for the node version, I would recommend upgrading. If you could provide that debug log file that NPM is pointing out (npm-debug.log) it might also help us to debug with you.

@FADHLyemen
Copy link
Author

I installed nodejs 14, please see below
image

now I run it in jupyter notebook with option proxy =TRUE

image

However can not access
http://127.0.0.1:8000/
image

did I miss any thing?

@keller-mark
Copy link
Member

This is correct. There is no web page to see at the 127.0.0.1:8000 URL, it is just the hostname of the web server where the raw data is being served from.

Also, the http://127.0.0.1:8000/ URL is local to the HPC where the Jupyter server is running (rather than your local machine). I will make an issue to change the log message to make it clearer about the context of that URL for the remote/HPC use cases.

@FADHLyemen
Copy link
Author

you make it easy for me.
Here what I want to do. I want to put my h5ad file in amazon s3 and visualize the data using vitessce. I want to send our collaborator a link so they can visualize the data without installing any thing at their end. is this an option in current version of vitessce? could you direct me to do it correctly
thank you

@ilan-gold
Copy link
Collaborator

@FADHLyemen Yes that is very doable. We have a notebook for doing just that, which starts with an h5ad file (that you download, so you may simply substitute your own), and then proceeds to set up the vitessce view configuration for it and upload to amazon s3.

@keller-mark
Copy link
Member

Please see these instructions to make sure that your AWS S3 bucket permissions allow accessing the data on the web http://beta.vitessce.io/docs/data-hosting/index.html#aws-s3

@FADHLyemen
Copy link
Author

Thank you, how to get the VITESSCE_S3_ACCESS_KEY_ID.txt and VITESSCE_S3_SECRET_ACCESS_KEY.txt

@FADHLyemen
Copy link
Author

@FADHLyemen
Copy link
Author

I still get some error, I appreciate your help
image

@keller-mark
Copy link
Member

You can replace the entire part after the equals sign with the string.

like this:

s3 = boto3.resource(
    service_name="s3",
    aws_access_key_id="my_access_key_id",
    aws_secret_access_key="my_secret_access_key"
)

@FADHLyemen
Copy link
Author

when I run
config_dict = vc.export(to='S3', s3=s3, bucket_name='kidneyscrnaseq')
I got below error:

boto3.exceptions.S3UploadFailedError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

I read this [solution]

and modify the bucket privacy to:

{
    "Version": "2012-10-17",
    "Id": "Policy12345",
    "Statement": [
        {
            "Sid": "Stmt6789",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::kidneyscrnaseq/*"
        }
    ]
}

it uploaded successfully.

now when I run webbrowser.open the cell is busy for long time. I do not know which link to send to collaborator and how to open the link.

image

@ilan-gold
Copy link
Collaborator

ilan-gold commented Jul 28, 2021

The string contents of the "Out[7]" cell are the url you can copy, paste, share, and navigate into a web browser

@FADHLyemen
Copy link
Author

I just got this for long time, I do not know what is going on:
image

@FADHLyemen
Copy link
Author

Now, it gives this empty page
image

@ilan-gold
Copy link
Collaborator

ilan-gold commented Jul 28, 2021

@keller-mark I am trying his URL but getting a 403. Please correct me if I'm wrong @keller-mark but I believe you @FADHLyemen need to add s3:ListBucket to your bucket actions. I think we had someone else have this problem where they got 403 not 404 - I think the documentation has been updated but not redeployed.

@ilan-gold
Copy link
Collaborator

Also s3:GetObjectAcl probably just to be safe....

@FADHLyemen
Copy link
Author

sorry, I should send you my URL:
http://vitessce.io/?url=data:,%7B%22version%22%3A+%221.0.2%22%2C+%22name%22%3A+%22Habib+et+al%22%2C+%22description%22%3A+%22COVID-19+Healthy+Donor+Brain%22%2C+%22datasets%22%3A+%5B%7B%22uid%22%3A+%22A%22%2C+%22name%22%3A+%22Brain%22%2C+%22files%22%3A+%5B%7B%22type%22%3A+%22cells%22%2C+%22fileType%22%3A+%22anndata-cells.zarr%22%2C+%22url%22%3A+%22https%3A%2F%2Fkidneyscrnaseq.s3.amazonaws.com%2FA%2F0%2Fanndata.zarr%22%2C+%22options%22%3A+%7B%22mappings%22%3A+%7B%22UMAP%22%3A+%7B%22key%22%3A+%22obsm%2FX_umap%22%2C+%22dims%22%3A+%5B0%2C+1%5D%7D%7D%7D%7D%2C+%7B%22type%22%3A+%22cell-sets%22%2C+%22fileType%22%3A+%22anndata-cell-sets.zarr%22%2C+%22url%22%3A+%22https%3A%2F%2Fkidneyscrnaseq.s3.amazonaws.com%2FA%2F0%2Fanndata.zarr%22%2C+%22options%22%3A+%5B%7B%22groupName%22%3A+%22Cell+Type%22%2C+%22setName%22%3A+%22obs%2FCellType%22%7D%5D%7D%2C+%7B%22type%22%3A+%22expression-matrix%22%2C+%22fileType%22%3A+%22anndata-expression-matrix.zarr%22%2C+%22url%22%3A+%22https%3A%2F%2Fkidneyscrnaseq.s3.amazonaws.com%2FA%2F0%2Fanndata.zarr%22%2C+%22options%22%3A+%7B%22matrix%22%3A+%22X%22%2C+%22matrixGeneFilter%22%3A+%22var%2Ftop_highly_variable%22%7D%7D%5D%7D%5D%2C+%22coordinationSpace%22%3A+%7B%22dataset%22%3A+%7B%22A%22%3A+%22A%22%7D%2C+%22embeddingType%22%3A+%7B%22A%22%3A+%22X_umap%22%7D%7D%2C+%22layout%22%3A+%5B%7B%22component%22%3A+%22scatterplot%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%2C+%22embeddingType%22%3A+%22A%22%7D%2C+%22x%22%3A+0.0%2C+%22y%22%3A+0.0%2C+%22w%22%3A+6.0%2C+%22h%22%3A+6.0%7D%2C+%7B%22component%22%3A+%22cellSets%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%7D%2C+%22x%22%3A+6.0%2C+%22y%22%3A+0.0%2C+%22w%22%3A+6.0%2C+%22h%22%3A+3.0%7D%2C+%7B%22component%22%3A+%22genes%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%7D%2C+%22x%22%3A+6.0%2C+%22y%22%3A+3.0%2C+%22w%22%3A+6.0%2C+%22h%22%3A+3.0%7D%2C+%7B%22component%22%3A+%22heatmap%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%7D%2C+%22x%22%3A+0%2C+%22y%22%3A+6.0%2C+%22w%22%3A+12%2C+%22h%22%3A+6.0%7D%5D%2C+%22initStrategy%22%3A+%22auto%22%7D

@keller-mark
Copy link
Member

Yes, sorry I thought it had been re-deployed.

https://github.com/vitessce/vitessce/blob/keller-mark/docs/docs/docs/data-hosting.md#bucket-policy

{
  "Version": "2012-10-17",
  "Id": "Policy12345",
  "Statement": [
    {
      "Sid": "Stmt6789",
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::foo-bar",
        "arn:aws:s3:::foo-bar/*"
      ]
    }
  ]
}

@FADHLyemen
Copy link
Author

@keller-mark if I did not include s3:PutObject" in action, I got access denied.
so below is my bucket policy

{
"Version": "2012-10-17",
"Id": "Policy12345",
"Statement": [
{
"Sid": "Stmt6789",
"Effect": "Allow",
"Principal": "",
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::kidneyscrnaseq",
"arn:aws:s3:::kidneyscrnaseq/
"
]
}
]
}

I rerun

s3 = boto3.resource(
    service_name='s3',
    aws_access_key_id='my code',
    aws_secret_access_key='my secret code',
)
config_dict = vc.export(to='S3', s3=s3, bucket_name='kidneyscrnaseq')

vitessce_url = "http://vitessce.io/?url=data:," + quote_plus(json.dumps(config_dict))

vitessce_url is

http://vitessce.io/?url=data:,%7B%22version%22%3A+%221.0.2%22%2C+%22name%22%3A+%22Habib+et+al%22%2C+%22description%22%3A+%22COVID-19+Healthy+Donor+Brain%22%2C+%22datasets%22%3A+%5B%7B%22uid%22%3A+%22A%22%2C+%22name%22%3A+%22Brain%22%2C+%22files%22%3A+%5B%7B%22type%22%3A+%22cells%22%2C+%22fileType%22%3A+%22anndata-cells.zarr%22%2C+%22url%22%3A+%22https%3A%2F%2Fkidneyscrnaseq.s3.amazonaws.com%2FA%2F0%2Fanndata.zarr%22%2C+%22options%22%3A+%7B%22mappings%22%3A+%7B%22UMAP%22%3A+%7B%22key%22%3A+%22obsm%2FX_umap%22%2C+%22dims%22%3A+%5B0%2C+1%5D%7D%7D%7D%7D%2C+%7B%22type%22%3A+%22cell-sets%22%2C+%22fileType%22%3A+%22anndata-cell-sets.zarr%22%2C+%22url%22%3A+%22https%3A%2F%2Fkidneyscrnaseq.s3.amazonaws.com%2FA%2F0%2Fanndata.zarr%22%2C+%22options%22%3A+%5B%7B%22groupName%22%3A+%22Cell+Type%22%2C+%22setName%22%3A+%22obs%2FCellType%22%7D%5D%7D%2C+%7B%22type%22%3A+%22expression-matrix%22%2C+%22fileType%22%3A+%22anndata-expression-matrix.zarr%22%2C+%22url%22%3A+%22https%3A%2F%2Fkidneyscrnaseq.s3.amazonaws.com%2FA%2F0%2Fanndata.zarr%22%2C+%22options%22%3A+%7B%22matrix%22%3A+%22X%22%2C+%22matrixGeneFilter%22%3A+%22var%2Ftop_highly_variable%22%7D%7D%5D%7D%5D%2C+%22coordinationSpace%22%3A+%7B%22dataset%22%3A+%7B%22A%22%3A+%22A%22%7D%2C+%22embeddingType%22%3A+%7B%22A%22%3A+%22X_umap%22%7D%7D%2C+%22layout%22%3A+%5B%7B%22component%22%3A+%22scatterplot%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%2C+%22embeddingType%22%3A+%22A%22%7D%2C+%22x%22%3A+0.0%2C+%22y%22%3A+0.0%2C+%22w%22%3A+6.0%2C+%22h%22%3A+6.0%7D%2C+%7B%22component%22%3A+%22cellSets%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%7D%2C+%22x%22%3A+6.0%2C+%22y%22%3A+0.0%2C+%22w%22%3A+6.0%2C+%22h%22%3A+3.0%7D%2C+%7B%22component%22%3A+%22genes%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%7D%2C+%22x%22%3A+6.0%2C+%22y%22%3A+3.0%2C+%22w%22%3A+6.0%2C+%22h%22%3A+3.0%7D%2C+%7B%22component%22%3A+%22heatmap%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%7D%2C+%22x%22%3A+0%2C+%22y%22%3A+6.0%2C+%22w%22%3A+12%2C+%22h%22%3A+6.0%7D%5D%2C+%22initStrategy%22%3A+%22auto%22%7D

which is still not working. did I miss any thing?

@ilan-gold
Copy link
Collaborator

In your view config, you have

"embeddingType": {
   "A": "X_umap"
}

which should be

"embeddingType": {
   "A": "UMAP"
}

because your anndata-cells loader looks like:

{
  "type": "cells",
  "fileType": "anndata-cells.zarr",
  "url": "https://kidneyscrnaseq.s3.amazonaws.com/A/0/anndata.zarr",
  "options": {
    "mappings": {
      "UMAP": {
        "key": "obsm/X_umap",
        "dims": [
          0,
          1
        ]
      }
    }
  }
}

i.e the mapping needs to match the embeddingType.

It looks like this is a bug in our example that you followed. So replace

scatterplot = vc.add_view(dataset, cm.SCATTERPLOT, mapping="X_umap")

with

scatterplot = vc.add_view(dataset, cm.SCATTERPLOT, mapping="UMAP")

Once I fix that I am able to see your dataset - sorry about that!

@FADHLyemen
Copy link
Author

Wow, it works, thank you @keller-mark and @ilan-gold for your patience.
I have two things. first this is a very long link, any suggestion to short it? second, do you have example of vitessce with umap for scrna-seq and image. I can not find this in your example. how to connect the h5ad with the image coordinates. thank you again

http://vitessce.io/?url=data:,%7B%22version%22%3A+%221.0.2%22%2C+%22name%22%3A+%22Habib+et+al%22%2C+%22description%22%3A+%22COVID-19+Healthy+Donor+Brain%22%2C+%22datasets%22%3A+%5B%7B%22uid%22%3A+%22A%22%2C+%22name%22%3A+%22Brain%22%2C+%22files%22%3A+%5B%7B%22type%22%3A+%22cells%22%2C+%22fileType%22%3A+%22anndata-cells.zarr%22%2C+%22url%22%3A+%22https%3A%2F%2Fkidneyscrnaseq.s3.amazonaws.com%2FA%2F0%2Fanndata.zarr%22%2C+%22options%22%3A+%7B%22mappings%22%3A+%7B%22UMAP%22%3A+%7B%22key%22%3A+%22obsm%2FX_umap%22%2C+%22dims%22%3A+%5B0%2C+1%5D%7D%7D%7D%7D%2C+%7B%22type%22%3A+%22cell-sets%22%2C+%22fileType%22%3A+%22anndata-cell-sets.zarr%22%2C+%22url%22%3A+%22https%3A%2F%2Fkidneyscrnaseq.s3.amazonaws.com%2FA%2F0%2Fanndata.zarr%22%2C+%22options%22%3A+%5B%7B%22groupName%22%3A+%22Cell+Type%22%2C+%22setName%22%3A+%22obs%2FCellType%22%7D%5D%7D%2C+%7B%22type%22%3A+%22expression-matrix%22%2C+%22fileType%22%3A+%22anndata-expression-matrix.zarr%22%2C+%22url%22%3A+%22https%3A%2F%2Fkidneyscrnaseq.s3.amazonaws.com%2FA%2F0%2Fanndata.zarr%22%2C+%22options%22%3A+%7B%22matrix%22%3A+%22X%22%2C+%22matrixGeneFilter%22%3A+%22var%2Ftop_highly_variable%22%7D%7D%5D%7D%5D%2C+%22coordinationSpace%22%3A+%7B%22dataset%22%3A+%7B%22A%22%3A+%22A%22%7D%2C+%22embeddingType%22%3A+%7B%22A%22%3A+%22UMAP%22%7D%7D%2C+%22layout%22%3A+%5B%7B%22component%22%3A+%22scatterplot%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%2C+%22embeddingType%22%3A+%22A%22%7D%2C+%22x%22%3A+0.0%2C+%22y%22%3A+0.0%2C+%22w%22%3A+6.0%2C+%22h%22%3A+6.0%7D%2C+%7B%22component%22%3A+%22cellSets%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%7D%2C+%22x%22%3A+6.0%2C+%22y%22%3A+0.0%2C+%22w%22%3A+6.0%2C+%22h%22%3A+3.0%7D%2C+%7B%22component%22%3A+%22genes%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%7D%2C+%22x%22%3A+6.0%2C+%22y%22%3A+3.0%2C+%22w%22%3A+6.0%2C+%22h%22%3A+3.0%7D%2C+%7B%22component%22%3A+%22heatmap%22%2C+%22coordinationScopes%22%3A+%7B%22dataset%22%3A+%22A%22%7D%2C+%22x%22%3A+0%2C+%22y%22%3A+6.0%2C+%22w%22%3A+12%2C+%22h%22%3A+6.0%7D%5D%2C+%22initStrategy%22%3A+%22auto%22%7D

@ilan-gold
Copy link
Collaborator

ilan-gold commented Jul 28, 2021

@FADHLyemen Yes it is completely possible. Perhaps you can open a discussion for this new issue and we can resolve the issue there - especially if you have any data to share, I'd be happy to create an example notebook as this seems to be an increasingly common request.

@ilan-gold
Copy link
Collaborator

As for the URL length, you can use a URL shorten-er like tinyurl

@ilan-gold
Copy link
Collaborator

In the meantime I am going to close this issue @FADHLyemen but please do open a discussion post!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants