-
Notifications
You must be signed in to change notification settings - Fork 0
/
Files_upload_and_Volumes.Rmd
1042 lines (794 loc) · 31.8 KB
/
Files_upload_and_Volumes.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
---
title: "File upload and Volumes reference guide for Seven Bridges API R Client"
date: "`r Sys.Date()`"
output:
rmarkdown::html_document:
toc: true
toc_float: true
toc_depth: 4
number_sections: false
theme: "flatly"
highlight: "textmate"
css: "sevenbridges.css"
vignette: >
%\VignetteEngine{knitr::rmarkdown}
%\VignetteIndexEntry{File upload and Volumes reference guide for Seven Bridges API R Client}
%\VignetteEncoding{UTF-8}
---
<a name="top"></a>
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
eval = FALSE
)
```
# File upload
Seven Bridges platforms provide a few different methods for data import:
- Import from FTP or HTTP with the web interface
- The file upload API that you can directly call with the `sevenbridges2`
package
- The command line uploader
- Import from cloud storage - Volume
- Import from a DRS server
In this chapter we will explain how you can use the `sevenbridges2` API library
to upload your files to the Platform.
Although it is more intuitive to have these operations available on the `File`
object, they are separated and stored directly on the authentication object
`Auth`, because there are a separate group of endpoints themselves.
## Upload single file
You can upload files from your local computer to the Platform using the
`upload()` method on your `Auth` object. The method allows you to upload only
a single file for now.
To upload a file, you should provide its full path on your local computer
as the `path` parameter.
To specify the upload destination for your file you can use either `project` or
`parent` parameter. These two parameters should not be used together.
* **project** - `Project` object or project ID.
* **parent** - `File` object (of type `Folder`) or its ID.
By calling the `upload()` method you are creating an upload job that by
default starts to run immediately.
If you don't want to start the job immediately, just set the `init` parameter
to `TRUE` in order to only initialize the object.
This upload job is wrapped into an object of the class `Upload` where you can
see its details and call other actions on it.
Let's initialize an upload job that will upload a file into a project:
```{r}
# Authenticate
a <- Auth$new(platform = "aws-us", token = "<your-token>")
# Get the desired project to upload to
destination_project <- a$projects$get(project = "<project_id>")
# Create upload job and set destination project
upload_job <- a$upload(
path = "/path/to/your/file.txt",
project = destination_project,
overwrite = TRUE,
init = TRUE
)
```
If you would like to upload your file into a folder, you need to set the
`parent` parameter:
```{r}
# Get destination folder object
destination_folder <- a$files$get(id = "<folder_id>")
up <- a$upload(
path = "/path/to/your/file.txt",
parent = destination_folder,
overwrite = TRUE,
init = TRUE
)
```
## Upload fields and operations
Since we have initialized the upload job, let's see which actions can we run.
### Print upload job
First, let's print the `Upload` object to see what the API returned as the
response.
```{r}
up$print()
```
```
── Upload ─────────────────────────────────────────────────────────────────────
• initialized: TRUE
• part_length: 1
• part_size: 33554432
• file_size: 232
• overwrite: FALSE
• filename: file.txt
• project: <username_or_division>/api-testing
• path: /path/to/your/file.txt
• upload_id: 4OvRx8Z9vghNoAUqsgYtNuM2IsiIM8kghhjgi7igu79HX9QKZpDEh5TZDrmhPxF
```
<div align="right"><a href="#top">top</a></div>
### File size, part size and part number/length
In the previous example we can see that the API returned the upload id and some
information about sizes.
First we see the `file_size` in bytes (232), which is the real size of the file.
File upload actually splits files into parts in the background; parts are then
being uploaded one by one or in parallel and then merged again on destination.
Each part can weigh a maximum of 5 GB, while the default `part_size` is
recommended and set to be 32MB (which is 33554432B in our example).
Lastly, number of parts or `part_length` field, is also an important measure.
Maximum number of parts can be 10.000.
Since users can control part size through the `part_size` parameter
in `upload()` function, they should be careful not to set a size that is too
small for very large files, so that total number of parts doesn’t exceed the
limit of maximum 10.000.
<div align="right"><a href="#top">top</a></div>
### Start upload
Call the `start()` method on the upload job object do start the upload process.
```{r}
# Start upload
up$start()
```
If you want to skip the step where you need to call the `start()` method
to start the actual upload process, just set the `init` parameter back to
`FALSE` when creating the upload job and the upload process will start right
away.
```{r}
# Create upload job and start it immediately
up <- a$upload(
path = "/path/to/your/file.txt",
project = destination_project,
overwrite = TRUE,
init = FALSE
)
```
<div align="right"><a href="#top">top</a></div>
### Get status information about the job
In order to track the progress of the job, you can call the `info()` method
on the upload object.
```{r}
# Get upload progress info
up$info()
```
Apart from basic information, the result will also provide the info on the
number of uploaded parts up to that moment.
<div align="right"><a href="#top">top</a></div>
## List all ongoing uploads
Going back to the authentication object, there are two more operations for
uploads manipulation.
One is the method `list_ongoing_uploads()` that allows you to see the list of
all ongoing upload processes.
```{r}
# List ongoing uploads
a$list_ongoing_uploads()
```
<div align="right"><a href="#top">top</a></div>
## Abort upload process
The other one is `abort()`.
You can abort any upload process using the `upload_abort()` method.
To do so, you need to provide the ID of a process within the `upload_id`
parameter.
```{r}
# Abort upload
a$abort_upload(upload_id = "<id_of_the_upload_process>")
```
**Note** that in practice, if you start a big upload job, your R session
will be blocked until this process is finished. This functionality is work in
progress but the idea is to not block your main session in the future,
while the upload is running.
For now, you can create another R session on your own and track the
progress of the upload job there.
<div align="right"><a href="#top">top</a></div>
# Volumes
Cloud storage providers come with their own interfaces, features, and
terminology. At a certain level, though, they all view resources as data objects
organized in repositories. Authentication and operations are commonly defined
on those objects and repositories, and while each cloud provider might call
these things different names and apply different parameters to them, their basic
behavior is the same.
Seven Bridges environments mediate access to these repositories using volumes.
A volume is associated with a particular cloud storage repository that you have
enabled Seven Bridges to read from (and, optionally, to write to).
Currently, volumes may be created using two types of cloud storage repositories:
Amazon Web Services' (AWS) S3 buckets and Google Cloud Storage (GCS) buckets.
A volume enables you to treat the cloud repository associated with it as
external storage. You can 'import' files from the volume to your Seven Bridges
environment to use them as inputs for computation. Similarly, you can write
files from the Seven Bridges environment to your cloud storage by 'exporting'
them to your volume.
Learn more about volumes on the [Seven Bridges Platform](https://docs.sevenbridges.com/docs/connecting-cloud-storage-overview), [CGC](https://docs.cancergenomicscloud.org/docs/connect-cloud-storage-overview),
[BDC](https://sb-biodatacatalyst.readme.io/docs/connect-cloud-storage-overview)
and [CAVATICA](https://docs.cavatica.org/reference/connect-cloud-storage-overview).
All volume related operations for querying volumes, fetching a single volume,
and creating volumes are grouped under `volumes` path (`Volumes`
resource class) on the authentication object.
When operating with a single volume, it is represented as an object of the
`Volume` class which stores all volume information returned from the API and
additional methods you are able to call directly on the volume, like volume
update, deactivation, listing content, volume members management etc.
**Note** that all operations with volumes require `advance_access` parameter
to be set to TRUE. In most of the volume operations it is pre-set to `TRUE` by
default.
## List volumes
You can list all volumes you've registered by calling the `volumes$query()`
method from the authentication object. The method doesn't have any additional
query parameters that could allow you to search for volumes by specific
criteria, except the ones that control the number of results returned using
`limit` and `offset` parameters.
```{r}
# Query volumes
a$volumes$query()
```
The result returned is the `Collection` object with pagination ability.
<div align="right"><a href="#top">top</a></div>
## Get single volume information
In order to retrieve information about a single volume of interest, you can get
it using the `volumes$get()` method using its id as parameter.
Volume ID is usually presented in the `<division_name>/<volume_name>` form for
Enterprise users, while for public program users it can be in the
`<volume_owner>/<volume_name>` form.
```{r}
# Get volume
a$volumes$get(id = "<volume_owner_or_division>/<volume_name>")
```
<div align="right"><a href="#top">top</a></div>
## Create volumes - AWS (S3) using IAM User authentication type
For creating volumes we have exposed several functions for different cloud
providers and authentication types:
* `create_s3_using_iam_user`: creates S3 volume using IAM User authentication
type
* `create_s3_using_iam_role`: creates S3 volume using IAM Role authentication
type
* `create_google_using_iam_user`: creates GC volume using IAM User
authentication type
* `create_google_using_iam_role`: creates GC volume using IAM Role
authentication type
* `create_azure`: creates Azure volume (only RO privileges allowed)
* `create_ali_oss`: creates AliCloud volume (only RO privileges allowed)
For each of the functions it is possible to provide parameters via path
(`from_path`) to a JSON file where all required fields should be listed.
Examples of use are shown below:
```{r}
# Create AWS volume using IAM User authentication type
aws_iam_user_volume <- a$volumes$create_s3_using_iam_user(
name = "my_new_aws_user_volume",
bucket = "<bucket-name>",
description = "AWS IAM User volume",
access_key_id = "<access-key>",
secret_access_key = "<secret-access-key>"
)
aws_iam_user_volume_from_path <- a$volumes$create_s3_using_iam_user(
from_path = "path/to/my/json/file.json"
)
# Create AWS volume using IAM Role authentication type
aws_iam_role_volume <- a$volumes$create_s3_using_iam_role(
name = "my_new_aws_role_volume",
bucket = "<bucket-name>",
description = "AWS IAM Role volume",
role_arn = "<role-arn-key>",
external_id = "<external-id>"
)
aws_iam_role_volume_from_path <- a$volumes$create_s3_using_iam_role(
from_path = "path/to/my/json/file.json"
)
# Create Google Cloud volume using IAM User authentication type
gc_iam_user_volume <- a$volumes$create_google_using_iam_user(
name = "my_new_gc_user_volume",
access_mode = "RW",
bucket = "<bucket-name>",
description = "GC IAM User volume",
client_email = "<client_email>",
private_key = "<private_key-string>"
)
gc_iam_user_volume_from_path <- a$volumes$create_google_using_iam_user(
from_path = "path/to/my/json/file.json"
)
# Create Google Cloud volume using IAM Role authentication type
# by passing configuration parameter as named list
gc_iam_role_volume <- a$volumes$create_google_using_iam_role(
name = "my_new_gc_role_volume",
access_mode = "RO",
bucket = "<bucket-name>",
description = "GC IAM Role volume",
configuration = list(
type = "<type-name>",
audience = "<audience-link>",
subject_token_type = "<subject_token_type>",
service_account_impersonation_url = "<service_account_impersonation_url>",
token_url = "<token_url>",
credential_source = list(
environment_id = "<environment_id>",
region_url = "<region_url>",
url = "<url>",
regional_cred_verification_url = "<regional_cred_verification_url>"
)
)
)
# Create Google Cloud volume using IAM Role authentication type
# by passing configuration parameter as string path to configuration file
gc_iam_role_volume_config_file <- a$volumes$create_google_using_iam_role(
name = "my_new_gc_role_volume_cnf_file",
access_mode = "RO",
bucket = "<bucket-name>",
description = "GC IAM Role volume - using config file",
configuration = "path/to/config/file.json"
)
# Create Google Cloud volume using IAM Role authentication type
# using from_path parameter
gc_iam_role_volume_from_path <- a$volumes$create_google_using_iam_role(
from_path = "path/to/full/config/file.json"
)
# Create Azure volume
azure_volume <- a$volumes$create_azure(
name = "my_new_azure_volume",
description = "Azure volume",
endpoint = "<endpoint>",
container = "<bucket-name",
storage_account = "<storage_account-name>",
tenant_id = "<tenant_id>",
client_id = "<client_id>",
client_secret = "<client_secret>",
resource_id = "<resource_id>"
)
azure_volume_from_path <- a$volumes$create_azure(
from_path = "path/to/my/json/file.json"
)
# Create Ali Cloud volume
ali_volume <- a$volumes$create_ali_oss(
name = "my_new_azure_volume",
description = "Ali volume",
endpoint = "<endpoint>",
bucket = "<bucket-name",
access_key_id = "<access_key_id>",
secret_access_key = "<secret_access_key>"
)
ali_volume_from_path <- a$volumes$create_ali_oss(
from_path = "path/to/my/json/file.json"
)
```
<div align="right"><a href="#top">top</a></div>
## Volume object operations
When you've created a new volume, you can notice it is represented as an object
of the `Volume` class. To preview all volume information, use the `print()`
method:
```{r}
# Print volume info
print(aws_iam_user_volume)
```
Within this volume you have the following operations available to
execute:
* `update`: update volume information
* `list_contents` : list volume content
* `get_file`: get single volume file info
* `deactivate` : deactivate volume
* `reactivate` : reactivate previously deactivated volume
* `list_members`: list all volume members
* `add_member`: add new volume member
* `remove_member`: remove volume member
* `get_member`: get a volume member information
* `modify_member_permissions`: modify member permissions on the volume
* `delete` : delete previously deactivated volume
* `reload` : reload volume object to sync information
* `list_imports`: list all imports from the specified volume
* `list_exports`: list all exports to the specified volume
### Update volume
You can update volume's `description`, `access_mode` and `service` information.
Please, consult our [API documentation](https://docs.sevenbridges.com/reference/update-a-volume-v2)
on how to use the `service` parameter.
```{r}
# If the volume is created with RO access mode and RO credential parameters,
# and now we want to change it to RW, we should also set proper credential
# parameters that are connected to the RW user on the bucket.
# If it's created with RW credentials, but access mode is set to RO, then no
# change is needed in the credentials parameters.
aws_iam_user_volume$update(
description = "Updated to RW",
access_mode = "RW",
service = list(
credentials = list(
access_key_id = "<access_key_id_for_rw>",
secret_access_key = "<secret_access_key_for_rw>",
)
)
)
```
<div align="right"><a href="#top">top</a></div>
### Reload volume
To keep your local `Volume` object up to date with the volume on the platform,
you can always call the `reload()` function:
```{r}
# Reload volume object
aws_iam_user_volume$reload()
```
<div align="right"><a href="#top">top</a></div>
### List volume's content
This operation lists all volume files in the root directory of the bucket,
unless the `parent` parameter is specified. In that case, it lists the content
of that directory on the bucket.
The output is a `VolumeContentCollection` collection object, that contains
two fields:
-`items` for storing a list of `VolumeFile` objects (files on the
volume) and
-`prefixes` for storing a list of `VolumePrefix` objects or folders on the
volume.
You can also specify the `limit` parameter to control the number of results
returned.
Same as `Collection` objects, here we also have pagination functions to
return either the next page of results or all results. However, backward
pagination is not available for volume contents.
Users can also navigate through pages of results by using the
`continuation token` parameter or `link` to fetch the next chunk of results.
If you use the `link` parameter, it will overwrite all other parameters if set,
since it already contains the `limit` and `continuation_token` info.
```{r}
# List all files in root bucket directory
content_collection <- aws_iam_user_volume$list_contents(limit = 20)
# Print collection
content_collection
# List all files from a specific directory on the bucket
folder_files_collection <- aws_iam_user_volume$list_contents(
prefix = "<directory_name>"
)
# Get the next group of results by setting the continuation token
content_collection <- aws_iam_user_volume$list_contents(
limit = 20,
continuation_token = "<continuation_token>"
)
# Preview volume files
content_collection$items
# Preview volume prefixes/folders
content_collection$prefixes
# Preview links
aws_iam_user_volume$links
# Get the next group of results by setting the link parameter
aws_iam_user_volume$list_contents(link = "<link_to_next_results>")
# Or use VolumeContentCollection object's next_page() method for this:
content_collection$next_page()
# You can also fetch all results with the all() method
content_collection$all()
```
<div align="right"><a href="#top">top</a></div>
### Volume files and prefixes
Volume files and prefixes are also treated as objects and they contain some
operations that can be called on them.
#### Get VolumeFile info
This operation returns a single volume file information. The input parameter
can be file's id which is represented as location on the bucket
(`location)`, or a link to that file resource. The link is a `href` field
of the desired file received from the response when returning a list of volume
contents with `list_contents()`.
Empty arguments are not allowed along with setting both parameters together.
```{r}
# Get single volume file info - by setting file_location
vol_file1 <- aws_iam_user_volume$get_file(
location = "<file_location_on_bucket>"
)
# Get single volume file info - by setting link
vol_file1 <- aws_iam_user_volume$get_file(link = "full/request/link/to/file")
```
<div align="right"><a href="#top">top</a></div>
#### Reload VolumeFile object
To keep your local VolumeFile object up to date with the volume file on the
platform, you can always call the `reload()` function:
```{r}
vol_file1$reload()
```
<div align="right"><a href="#top">top</a></div>
#### Get VolumePrefix info
Unfortunately we don't have a separate operation to fetch only prefixes on the
volume, therefore, we can get its prefixes only by using the `list_contents()`
operation and look for the `prefixes` field in the returned
`VolumeContentCollection` object.
```{r}
# List content
volume_content <- aws_iam_user_volume$list_contents()
# Extract prefixes
volume_prefixes <- volume_content$prefixes
# Select one of the volume folders to list its content
volume_folder <- volume_prefixes[[1]]
# Print volume prefix information
volume_folder$print()
```
You can also list the content of a volume prefix/folder on the volume, by
calling `list_contents()` directly on the `VolumePrefix` object.
```{r}
## Select one of the volume folders to list its content
volume_folder <- volume_prefixes[[1]]
# List content
volume_folder_content <- volume_folder$list_contents()
```
<div align="right"><a href="#top">top</a></div>
### List volume members
In order to fetch members of one volume or a specific member by its username,
you can use `list_members()` and `get_member()` operations:
```{r}
# List volume members
aws_iam_user_volume$list_members() # limit = 2
# Get single member
aws_iam_user_volume$get_member(user = "<member-username>")
```
<div align="right"><a href="#top">top</a></div>
### Remove members
Volume admins can remove volume members by providing its username or object of
the `Member` class to the `remove_member()` function:
```{r}
# Remove member
aws_iam_user_volume$remove_member("<member-username>")
# Remove member using the Member object
members <- aws_iam_user_volume$list_members()
aws_iam_user_volume$remove_member(members$items[[3]])
```
<div align="right"><a href="#top">top</a></div>
### Adding new members
The function for adding new members to the volume can accept a `Member` object
(for example used in a project) or its username.
```{r}
# Add member via username
aws_iam_user_volume$add_member(user = "<member-username>", permissions = list(
read = TRUE, copy = TRUE, write = FALSE, admin = FALSE
))
# Add member via Member object
aws_iam_user_volume$add_member(
user = Member$new(
username = "<member-username>",
id = "<member-username>"
),
permissions = list(
read = TRUE, copy = TRUE, write = FALSE,
admin = FALSE
)
)
```
<div align="right"><a href="#top">top</a></div>
### Modifying a member's permissions
Users can modify specific member's permissions on the volume by providing the
privileges they want to change:
```{r}
# Modify member permissions
aws_iam_user_volume$modify_member_permissions(
user = "<member-username>",
permissions = list(write = TRUE)
)
```
<div align="right"><a href="#top">top</a></div>
### Deactivate and reactivate the volume
Once deactivated, you cannot import from, export to, or browse within a volume.
As such, the content of the files imported from this volume will no longer be
accessible on the platform. However, you can update the volume and manage
members.
Note that you cannot deactivate the volume if you have running imports or
exports unless you force the operation using the query parameter `force = TRUE`.
**Note** that to delete a volume, first you must deactivate it and delete all
files which have been imported from the volume to the platform.
To reactivate the volume, just use the `reactivate()` function.
```{r}
# Deactivate volume
aws_iam_user_volume$deactivate()
# Reactivate volume
aws_iam_user_volume$reactivate()
```
<div align="right"><a href="#top">top</a></div>
### Delete volume
To be able to delete a volume, you first need to deactivate it and then delete
all files on the Platform that were previously imported from the volume.
```{r}
# Deactivate volume
aws_iam_user_volume$deactivate()
# Delete volume
aws_iam_user_volume$delete()
```
<div align="right"><a href="#top">top</a></div>
# Imports
Creating and connecting volumes to the Platform allows you to import your
files/folders from a cloud bucket to the Platform. Imports operations are
related to volumes, but in the API they are separated under `/imports`
endpoints, so in our library they are also grouped under imports path on the
authentication object (`Imports` resource class).
A single import job is represented as an `Import` class object containing
information about which file/folder has been or is being imported, from which
volume, to which project/folder on the platform, import start and finish time,
status of the job, logs etc.
## List volume imports
To preview and query all import jobs you've created use the `query()` function
on `Auth$imports` path:
```{r}
# List imports
all_imports <- a$imports$query()
# Limit results to 5
imp_limit5 <- a$imports$query(limit = 5)
# Load next page of 5 results
imp_limit5$next_page(advance_access = TRUE)
# Load all results at once until last page
imp_limit5$all(advance_access = TRUE)
```
It is possible to use some query parameters as different criteria for filtering
results like `volume`, `project`, `state` etc:
```{r}
# List imports with state being RUNNING or FAILED
imp_states <- auth$imports$query(state = c("RUNNING", "FAILED"))
# List imports to the specific project
imp_project <- auth$imports$query(project = "<project_id>")
```
Listing imports is also available within `Project` and `Volume` objects, where
resulting imports are related to the specific project or volume where they're
called from.
```{r}
## Get the volume from which you want to list all imports
vol1 <- auth$volumes$get(id = "<volumes_owner_or_division>/<volume-name>")
vol1$list_imports()
## Get the project object for which you want to list imports
test_proj <- auth$projects$get("<project_id>")
test_proj$list_imports()
```
<div align="right"><a href="#top">top</a></div>
## Get a single import job
Similar to other resource classes, the `get()` method will return a single
import job object when provided with a job id.
```{r}
# Get single import
imp_obj <- a$imports$get(id = "<import_job_id>")
```
<div align="right"><a href="#top">top</a></div>
## Submit new import - import a volume file into the project
In order to import volume files into a project, users can use the
`submit_import()` method from the `Auth$imports` path, or directly on the
selected `VolumeFile` object (file they want to import) where this function is
also available.
```{r}
## First, get the volume you want to import files from
vol1 <- a$volumes$get(id = "<volume_owner_or_division>/<volume_name>")
## Then, get the project object/id where you want to import files
test_proj <- a$projects$get("<project_id>")
## List all volume files on the volume
vol1_content <- vol1$list_contents()
## Select one of the volume files
volume_file_import <- vol1_content$items[[3]]
## Perform a file import
imp_job1 <- a$imports$submit_import(
source_location = volume_file_import,
destination_project = test_proj,
autorename = TRUE
)
# Alternatively you can also call import() directly on the VolumeFile object
imp_job1 <- volume_file_import$import(
destination_project = test_proj,
autorename = TRUE
)
```
Preview import job details with the `print()` method:
```{r}
# Print Import object
print(imp_job1)
```
You can also import folders from the volume into the project, with the option to
preserve or not to preserve folder structure:
```{r}
# Select one of the volume folders to import
volume_folder_import <- vol1_content$prefixes[[1]]
# Perform a folder import
imp_job2 <- a$imports$submit_import(
source_location = volume_folder_import,
destination_project = test_proj,
overwrite = TRUE,
preserve_folder_structure = TRUE
)
# Alternatively you can also call import() directly on the VolumePrefix object
imp_job2 <- volume_folder_import$import(
destination_project = test_proj,
overwrite = TRUE,
preserve_folder_structure = TRUE
)
# Print Import object
print(imp_job2)
```
<div align="right"><a href="#top">top</a></div>
## Reload import job
In order to refresh the import job object and get the up to date info
about its state, you can always call the `reload()` function:
```{r}
# Reload import object
imp_job1$reload()
```
<div align="right"><a href="#top">top</a></div>
# Exports
Exports are the actions of exporting your files from the Platform into a
cloud bucket represented as a volume.
Export operations are also related to volumes, but in the API they are
separated under `/exports` endpoints, so in our library they are also grouped
under the `exports` path on the authentication object (`Exports` resource
class).
A single export job is represented as an `Export` class object containing
information about a file has been or is being exported, from which
project/folder on the Platform, to which volume, export start and finish time,
status of the job, logs etc.
## List file exports to the volumes
Users can preview and query all export jobs they've created for the purpose of
exporting their files from the Platform into a cloud bucket using volumes.
The output is a `Collection` object storing a list of exports in its `items`
field and providing pagination options.
```{r}
# List exports
all_exports <- a$exports$query()
# Limit results to 5
exp_limit5 <- a$exports$query(limit = 5)
# Load next page of 5 results
exp_limit5$next_page(advance_access = TRUE)
# List all results until last page
exp_limit5$all()
```
It is possible to use some query parameters as different criteria for filtering
results like `volume`, `state` etc:
```{r}
# List exports with status RUNNING or FAILED
exp_states <- a$exports$query(state = c("RUNNING", "FAILED"))
# List exports into a specific volume
exp_volume <- a$exports$query(
volume = "<volume_owner_or_division>/<volume_name>" # volume object or id
)
```
Listing exports is also available within `Volume` objects, where results
contain all files exported to the specific volume they're being called from.
```{r}
# Get the volume for which you want to list all exports
vol1 <- a$volumes$get(id = "<volume_owner_or_division>/<volume_name>")
# List exports
vol1$list_exports()
```
<div align="right"><a href="#top">top</a></div>
## Get a single export job
Similar to other resource classes, the `get()` method will return a single
export job object when provided with job id.
```{r}
# Get a single export
exp_obj <- auth$exports$get(id = "<export_job_id>")
```
<div align="right"><a href="#top">top</a></div>
## Submit a new export - export a file from the platform to a volume
In order to export platform files into volumes, users can use the
`submit_export()` method from the `auth$exports` path, or directly on the
selected `File` object (file they want to export) where this function is also
available.
```{r}
# First, get the volume you want to export files to
vol1 <- a$volumes$get(id = "<volume_owner_or_division>/<volume_name>")
# Get the File object/id you want to export from the platform
test_file <- a$files$get("<file_id>")
# Perform a file export
exp_job1 <- a$exports$submit_export(
source_file = test_file,
destination_volume = vol1,
destination_location = "new_volume_file.txt" # new name
)
```
Preview export job details with the `print()` method:
```{r}
# Print export job info
print(exp_job1)
```
Bear in mind that **folders export** from the platform to volumes is
**not possible** with this function. For such cases (or export of multiple
files) it is better to use bulk actions that will be added to the package soon.