Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tiledb_object_walk() column contents has to be swapped #683

Closed
cgiachalis opened this issue Mar 30, 2024 · 13 comments
Closed

tiledb_object_walk() column contents has to be swapped #683

cgiachalis opened this issue Mar 30, 2024 · 13 comments

Comments

@cgiachalis
Copy link
Contributor

In short, in the output dataframe TYPE column holds the uris and URI column the types. Docs states a dataframe with object type, object uri string columns, so the contents has to be swapped.

library(tiledb)

root_uri <- tempdir()
uri <- paste0(root_uri, "\\temp")
fromDataFrame(iris, uri)
res <- tiledb_object_walk(root_uri)

res$URI
 [1] "ARRAY"

res$TYPE
 [1] "file:///C:/Users/<.....>/Temp/RtmpEPaF1U/temp"

The root cause is in libtiledb_object_walk:

TileDB-R/src/libtiledb.cpp

Lines 4327 to 4328 in d796865

return Rcpp::DataFrame::create(Rcpp::Named("TYPE") = uris,
Rcpp::Named("URI") = types);

If we swap uris with types we get the desired result:

 res$URI
[1] "file:///C:/Users/<....>/Temp/Rtmpq0MVLA/temp"

 res$TYPE
[1] "ARRAY"

Let me know if you wish to see a PR.
@eddelbuettel
Copy link
Member

Another excellent catch.

Let me check with the documentation to see what C++, Python, ... do so that I can align accordingly.

@eddelbuettel
Copy link
Member

Not entirely clear from looking at https://docs.tiledb.com/main/how-to/object-management so I think i wil just do a quick clean-up along the lines you suggested.

It'll also look nicer once TYPE is short. Quick before with a standard single-cell data set I have here.

> tiledb::tiledb_object_walk("/tmp/tiledb/soco")
                                                                                   TYPE   URI
1                                              file:///tmp/tiledb/soco/pbmc3k_processed GROUP
2                                           file:///tmp/tiledb/soco/pbmc3k_processed/ms GROUP
3                                       file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA GROUP
4                                     file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/X GROUP
5                                file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/X/data ARRAY
6                                  file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsm GROUP
7                  file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsm/X_draw_graph_fr ARRAY
8                            file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsm/X_pca ARRAY
9                           file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsm/X_tsne ARRAY
10                          file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsm/X_umap ARRAY
11                                 file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsp GROUP
12                  file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsp/connectivities ARRAY
13                       file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsp/distances ARRAY
14                                  file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns GROUP
15                       file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/draw_graph GROUP
16                file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/draw_graph/params GROUP
17   file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/draw_graph/params/random_state ARRAY
18                          file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/louvain GROUP
19                   file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/louvain/params GROUP
20      file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/louvain/params/random_state ARRAY
21        file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/louvain/params/resolution ARRAY
22                   file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/louvain_colors ARRAY
23                        file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/neighbors GROUP
24                 file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/neighbors/params GROUP
25     file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/neighbors/params/n_neighbors ARRAY
26                              file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/pca GROUP
27                     file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/pca/variance ARRAY
28               file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/pca/variance_ratio ARRAY
29                file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/rank_genes_groups GROUP
30         file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/rank_genes_groups/params GROUP
31 file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/rank_genes_groups/params/use_raw ARRAY
32                                  file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/var ARRAY
33                                 file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/varm GROUP
34                             file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/varm/PCs ARRAY
35                                      file:///tmp/tiledb/soco/pbmc3k_processed/ms/raw GROUP
36                                    file:///tmp/tiledb/soco/pbmc3k_processed/ms/raw/X GROUP
37                               file:///tmp/tiledb/soco/pbmc3k_processed/ms/raw/X/data ARRAY
38                                  file:///tmp/tiledb/soco/pbmc3k_processed/ms/raw/var ARRAY
39                                         file:///tmp/tiledb/soco/pbmc3k_processed/obs ARRAY
40                                               file:///tmp/tiledb/soco/subset_100_100 GROUP
41                                            file:///tmp/tiledb/soco/subset_100_100/ms GROUP
42                                        file:///tmp/tiledb/soco/subset_100_100/ms/RNA GROUP
43                                      file:///tmp/tiledb/soco/subset_100_100/ms/RNA/X GROUP
44                                 file:///tmp/tiledb/soco/subset_100_100/ms/RNA/X/data ARRAY
45                                    file:///tmp/tiledb/soco/subset_100_100/ms/RNA/var ARRAY
46                                           file:///tmp/tiledb/soco/subset_100_100/obs ARRAY
>

@cgiachalis
Copy link
Contributor Author

Excellent! Thanks for your prompt response.

@eddelbuettel
Copy link
Member

After the fix:

> tiledb::tiledb_object_walk("/tmp/tiledb/soco")
    TYPE                                                                                  URI
1  GROUP                                             file:///tmp/tiledb/soco/pbmc3k_processed
2  GROUP                                          file:///tmp/tiledb/soco/pbmc3k_processed/ms
3  GROUP                                      file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA
4  GROUP                                    file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/X
5  ARRAY                               file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/X/data
6  GROUP                                 file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsm
7  ARRAY                 file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsm/X_draw_graph_fr
8  ARRAY                           file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsm/X_pca
9  ARRAY                          file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsm/X_tsne
10 ARRAY                          file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsm/X_umap
11 GROUP                                 file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsp
12 ARRAY                  file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsp/connectivities
13 ARRAY                       file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/obsp/distances
14 GROUP                                  file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns
15 GROUP                       file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/draw_graph
16 GROUP                file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/draw_graph/params
17 ARRAY   file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/draw_graph/params/random_state
18 GROUP                          file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/louvain
19 GROUP                   file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/louvain/params
20 ARRAY      file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/louvain/params/random_state
21 ARRAY        file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/louvain/params/resolution
22 ARRAY                   file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/louvain_colors
23 GROUP                        file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/neighbors
24 GROUP                 file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/neighbors/params
25 ARRAY     file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/neighbors/params/n_neighbors
26 GROUP                              file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/pca
27 ARRAY                     file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/pca/variance
28 ARRAY               file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/pca/variance_ratio
29 GROUP                file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/rank_genes_groups
30 GROUP         file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/rank_genes_groups/params
31 ARRAY file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/uns/rank_genes_groups/params/use_raw
32 ARRAY                                  file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/var
33 GROUP                                 file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/varm
34 ARRAY                             file:///tmp/tiledb/soco/pbmc3k_processed/ms/RNA/varm/PCs
35 GROUP                                      file:///tmp/tiledb/soco/pbmc3k_processed/ms/raw
36 GROUP                                    file:///tmp/tiledb/soco/pbmc3k_processed/ms/raw/X
37 ARRAY                               file:///tmp/tiledb/soco/pbmc3k_processed/ms/raw/X/data
38 ARRAY                                  file:///tmp/tiledb/soco/pbmc3k_processed/ms/raw/var
39 ARRAY                                         file:///tmp/tiledb/soco/pbmc3k_processed/obs
40 GROUP                                               file:///tmp/tiledb/soco/subset_100_100
41 GROUP                                            file:///tmp/tiledb/soco/subset_100_100/ms
42 GROUP                                        file:///tmp/tiledb/soco/subset_100_100/ms/RNA
43 GROUP                                      file:///tmp/tiledb/soco/subset_100_100/ms/RNA/X
44 ARRAY                                 file:///tmp/tiledb/soco/subset_100_100/ms/RNA/X/data
45 ARRAY                                    file:///tmp/tiledb/soco/subset_100_100/ms/RNA/var
46 ARRAY                                           file:///tmp/tiledb/soco/subset_100_100/obs
>

Need to now ponder if this warrants an 'interface change' (I lean towards no) and what it may do to tests. Apparently nothing (good !) but I seem to have left the last PR with a missing re-roxygenize run I may wrap into this.

@eddelbuettel
Copy link
Member

I will make this two PRs just to be cleaner.

@cgiachalis
Copy link
Contributor Author

Nice. Is it good time to ask if labels are coming to the R client? :)

@cgiachalis
Copy link
Contributor Author

Oh..you might want to update the LICENSE year

TileDB-R/LICENSE

Line 1 in d796865

YEAR: 2017-2021

@eddelbuettel eddelbuettel mentioned this issue Mar 30, 2024
Merged
@eddelbuettel
Copy link
Member

Wow. Another one.

@eddelbuettel
Copy link
Member

Nice. Is it good time to ask if labels are coming to the R client? :)

Err, what labels? Am I behind some API offerings not covered?

@cgiachalis
Copy link
Contributor Author

https://tiledb-inc-tiledb.readthedocs-hosted.com/projects/tiledb-py/en/stable/python-api.html#tiledb.ArraySchema

https://tiledb-inc-tiledb.readthedocs-hosted.com/projects/tiledb-py/en/stable/python-api.html#tiledb.Dim.create_label_schema

@eddelbuettel
Copy link
Member

eddelbuettel commented Mar 30, 2024 •
edited

"Interesting"

I don't see it at https://docs.tiledb.com/main/how-to/arrays/creating-arrays/creating-the-array-schema but will poke.

@cgiachalis
Copy link
Contributor Author

Might be still experimental?

@eddelbuettel
Copy link
Member

Apparently, there is a header dimension_label_experimental.h. Will peruse / discuss. Thanks for the heads-up!

@eddelbuettel eddelbuettel mentioned this issue Apr 16, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@eddelbuettel @cgiachalis