Skip to content

Commit

Permalink
Support enabling active sync in Dataproc init action
Browse files Browse the repository at this point in the history
pr-link: #11403
change-id: cid-13676d0913f228f4d9d512ec6882c1fdeb871697
  • Loading branch information
madanadit committed May 8, 2020
1 parent 775e84c commit a0efcd9
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 2 deletions.
15 changes: 14 additions & 1 deletion docs/en/cloud/Google-Dataproc.md
Expand Up @@ -58,10 +58,23 @@ alluxio_site_properties="fs.gcs.accessKeyId=<my_access_key>;fs.gcs.secretAccessK
The Alluxio deployment on Google Dataproc can customized for more complex scenarios by passing
additional metadata labels to the `gcloud clusters create` command.
{% accordion download %}
{% collapsible Enable Active Sync on HDFS Paths %}
[Active Sync]({{ '/en/core-services/Unified-Namespace.html#metadata-active-sync-for-hdfs' | relativize_url}})
can be enabled on paths in Alluxio for a root HDFS mount point using the metadata key
`alluxio_sync_list`.
Specify a list of paths in Alluxio delimited using `;`.
```console
...
--metadata \
alluxio_sync_list="/tmp;/user/hadoop",\
...
```
{% endcollapsible %}

{% collapsible Download Additional Files %}
Additional files can be downloaded into the Alluxio installation directory at `/opt/alluxio/conf`
using the metadata key `alluxio_download_files_list`.
Specify `http(s)` or `gs` uris delimited using `;.`
Specify `http(s)` or `gs` uris delimited using `;`.
```console
...
--metadata \
Expand Down
12 changes: 11 additions & 1 deletion integration/dataproc/alluxio.sh
Expand Up @@ -105,6 +105,15 @@ start_alluxio() {
if [[ "${ROLE}" == "Master" ]]; then
${ALLUXIO_HOME}/bin/alluxio formatMaster
systemctl restart alluxio-master alluxio-job-master

local -r sync_list=$(/usr/share/google/get_metadata_value attributes/alluxio_sync_list || true)
local path_delimiter=";"
if [[ "${sync_list}" ]]; then
IFS="${path_delimiter}" read -ra paths <<< "${sync_list}"
for path in "${paths[@]}"; do
${ALLUXIO_HOME}/bin/alluxio fs startSync ${path}
done
fi
else
if [[ $(get_alluxio_property alluxio.worker.tieredstore.level0.alias) == "MEM" ]]; then
${ALLUXIO_HOME}/bin/alluxio-mount.sh SudoMount local
Expand Down Expand Up @@ -312,7 +321,8 @@ configure_alluxio() {
append_alluxio_property alluxio.master.security.impersonation.client.users "*"
append_alluxio_property alluxio.master.security.impersonation.client.groups "*"
append_alluxio_property alluxio.security.login.impersonation.username "_NONE_"
append_alluxio_property alluxio.security.authorization.permission.enabled "false"
append_alluxio_property alluxio.security.authorization.permission.enabled "true"
append_alluxio_property alluxio.user.rpc.retry.max.duration "10min"
local -r site_properties=$(/usr/share/google/get_metadata_value attributes/alluxio_site_properties || true)
local property_delimiter=";"
if [[ "${site_properties}" ]]; then
Expand Down

0 comments on commit a0efcd9

Please sign in to comment.