Problem checking out nodes using repo like /oak:index with escaped chars like :

[kitarek]

The issue summary might need to be re-qualified depending on investigation.
The current hypothesis I have is that there are troubles syncing (possibly in both ways) special JCR paths like /oak:index that are escaped by VLT. It seems to be related to #5

My version

+ VERSION=1.5-beta
+ server=http://localhost:4502
+ credentials=admin:admin
+ force=false
+ quiet=false
+ packmgr=/crx/packmgr/service/.json
+ packageGroup=tmp/repo
++ basename /Users/kitarek/bin/repo
+ readonly PROGNAME=repo
+ PROGNAME=repo
+ trap 'cleanup 130' SIGINT SIGTERM SIGHUP SIGQUIT
+ '[' 1 -eq 0 ']'
+ [[ --version == \-\h ]]
+ [[ --version == \-\-\h\e\l\p ]]
+ [[ --version == \h\e\l\p ]]
+ [[ --version == \-\v\e\r\s\i\o\n ]]
+ [[ --version == \-\-\v\e\r\s\i\o\n ]]
+ echo 'repo version 1.5-beta'
repo version 1.5-beta
+ exit

Steps to reproduce on local AEM instance

0. Add set -x line to repo tool to see debug logs for bash
1. mkdir repo-oak-index
2. cd repo-oak-index/
3. repo checkout '/oak:index'
4. However, repo checkout '/apps' works correctly so in general the problem is not related to local environment.

The output of 3. is:

+ server=http://localhost:4502
+ credentials=admin:admin
+ force=false
+ quiet=false
+ packmgr=/crx/packmgr/service/.json
+ packageGroup=tmp/repo
++ basename /Users/kitarek/bin/repo
+ readonly PROGNAME=repo
+ PROGNAME=repo
+ trap 'cleanup 130' SIGINT SIGTERM SIGHUP SIGQUIT
+ '[' 2 -eq 0 ']'
+ [[ checkout == \-\h ]]
+ [[ checkout == \-\-\h\e\l\p ]]
+ [[ checkout == \h\e\l\p ]]
+ [[ checkout == \-\v\e\r\s\i\o\n ]]
+ [[ checkout == \-\-\v\e\r\s\i\o\n ]]
+ '[' -t 1 ']'
+ isatty=true
+ action=checkout
+ shift
+ '[' checkout == st ']'
+ [[ checkout == \c\h\e\c\k\o\u\t ]]
+ [[ 1 -eq 0 ]]
+ path=/Users/kitarek/repo-oak-index
+ '[' 1 -gt 0 ']'
+ [[ /oak:index == --* ]]
+ [[ /oak:index == -* ]]
+ path=/oak:index
+ shift
+ '[' 0 -gt 0 ']'
+ '[' checkout == checkout ']'
+ filter=/oak:index
+ [[ /Users/kitarek/repo-oak-index == */jcr_root* ]]
+ '[' -e jcr_root ']'
+ mkdir jcr_root
+ rootpath=/Users/kitarek/repo-oak-index/jcr_root
+ echo 'checking out into new /Users/kitarek/repo-oak-index/jcr_root'
checking out into new /Users/kitarek/repo-oak-index/jcr_root
+ echo

+ path=/Users/kitarek/repo-oak-index/jcr_root/oak:index
+ mkdir -p /Users/kitarek/repo-oak-index/jcr_root/oak:index
+ action=get
+ [[ /oak:index == '' ]]
+ [[ /oak:index == \/ ]]
+ fromVlt=false
++ find_up /Users/kitarek/repo-oak-index .repo
+++ abspath /Users/kitarek/repo-oak-index
+++ '[' -d /Users/kitarek/repo-oak-index ']'
++++ cd /Users/kitarek/repo-oak-index
++++ pwd
+++ echo /Users/kitarek/repo-oak-index
++ dir=/Users/kitarek/repo-oak-index
++ '[' -n /Users/kitarek/repo-oak-index ']'
++ '[' -e /Users/kitarek/repo-oak-index/.repo ']'
++ dir=/Users/kitarek
++ '[' -n /Users/kitarek ']'
++ '[' -e /Users/kitarek/.repo ']'
++ dir=/Users
++ '[' -n /Users ']'
++ '[' -e /Users/.repo ']'
++ dir=
++ '[' -n '' ']'
+ repoCfg=
+ '[' -f '' ']'
+ '[' -f /Users/kitarek/repo-oak-index/jcr_root/.vlt ']'
+ server=http://localhost:4502
+ credentials=admin:admin
+ force=false
+ quiet=false
+ false
+ verbose=true
+ inverseDiff=false
+ '[' get == localdiff ']'
+ '[' get == serverdiff ']'
+ false
+ pathDirname=/Users/kitarek/repo-oak-index/jcr_root
+ filterDirname=
+ pathBasename=oak:index
+ filterBasename=oak:index
+ '[' -d /Users/kitarek/repo-oak-index/jcr_root/oak:index ']'
+ humanFilter='/oak:index/*'
+ '[' get == put ']'
+ '[' get == get ']'
+ print 'downloading /oak:index/* from http://localhost:4502'
+ true
+ echo 'downloading /oak:index/* from http://localhost:4502'
downloading /oak:index/* from http://localhost:4502
+ packageName=-oak:index
+ packageName=repo-oakindex
++ date +%s
+ packageVersion=1519992982
++ mktemp -d -t repo.XXX
+ tmpDir=/var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs
+ excludes=/var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/.excludes
+ write_excludes /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/.excludes
+ cat
+ cat /Users/kitarek/repo-oak-index/jcr_root/.vltignore
+ cat /Users/kitarek/repo-oak-index/jcr_root/.repoignore
++ find /Users/kitarek/repo-oak-index/jcr_root/oak:index -name .vltignore
+ create_pkg_meta_inf /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs /oak:index tmp/repo repo-oakindex 1519992982
+ local base=/var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs
++ filesystem_to_jcr /oak:index
++ local filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
+++ urldecode /oak:index
+++ printf %b /oak:index
++ filter=/oak:index
++ echo /oak:index
+ local filter=/oak:index
+ local pkgGroup=tmp/repo
+ local pkgName=repo-oakindex
+ local pkgVersion=1519992982
+ mkdir -p /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/META-INF/vault
+ mkdir -p /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/jcr_root
+ cat
++ to_xml /oak:index
++ local name=/oak:index
++ name=/oak:index
++ name=/oak:index
++ name=/oak:index
++ name=/oak:index
++ name=/oak:index
++ echo /oak:index
+ cat
++ to_xml repo-oakindex
++ local name=repo-oakindex
++ name=repo-oakindex
++ name=repo-oakindex
++ name=repo-oakindex
++ name=repo-oakindex
++ name=repo-oakindex
++ echo repo-oakindex
++ to_xml 1519992982
++ local name=1519992982
++ name=1519992982
++ name=1519992982
++ name=1519992982
++ name=1519992982
++ name=1519992982
++ echo 1519992982
++ to_xml tmp/repo
++ local name=tmp/repo
++ name=tmp/repo
++ name=tmp/repo
++ name=tmp/repo
++ name=tmp/repo
++ name=tmp/repo
++ echo tmp/repo
+ pkg=tmp/repo/repo-oakindex-1519992982.zip
+ '[' get == put ']'
+ [[ get == \g\e\t ]]
+ build_zip /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs
+ pushd /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs
+ zip -q -r pkg.zip .
+ popd
+ zipfile=/var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/pkg.zip
+ upload_pkg /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/pkg.zip
+ pkg_mgr_http upload '-F package=@/var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/pkg.zip -F force=true http://localhost:4502/crx/packmgr/service/.json?cmd=upload'
++ curl_http '-F package=@/var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/pkg.zip -F force=true http://localhost:4502/crx/packmgr/service/.json?cmd=upload'
++ curl -u admin:admin -f -s -S -F package=@/var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/pkg.zip -F force=true 'http://localhost:4502/crx/packmgr/service/.json?cmd=upload'
+ out='{"success":true,"msg":"Package uploaded","path":"/etc/packages/tmp/repo/repo-oakindex-1519992982.zip"}'
+ [[ {"success":true,"msg":"Package uploaded","path":"/etc/packages/tmp/repo/repo-oakindex-1519992982.zip"} != *\"success\":true* ]]
+ build_pkg tmp/repo/repo-oakindex-1519992982.zip
+ pkg_mgr_http build '-X POST http://localhost:4502/crx/packmgr/service/.json/etc/packages/tmp/repo/repo-oakindex-1519992982.zip?cmd=build'
++ curl_http '-X POST http://localhost:4502/crx/packmgr/service/.json/etc/packages/tmp/repo/repo-oakindex-1519992982.zip?cmd=build'
++ curl -u admin:admin -f -s -S -X POST 'http://localhost:4502/crx/packmgr/service/.json/etc/packages/tmp/repo/repo-oakindex-1519992982.zip?cmd=build'
+ out='{"success":true,"msg":"Package built"}'
+ [[ {"success":true,"msg":"Package built"} != *\"success\":true* ]]
+ rm -rf /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/META-INF /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/jcr_root /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/pkg.zip
+ download_pkg tmp/repo/repo-oakindex-1519992982.zip /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/pkg.zip
+ curl_http '-o /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/pkg.zip http://localhost:4502/etc/packages/tmp/repo/repo-oakindex-1519992982.zip'
+ curl -u admin:admin -f -s -S -o /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/pkg.zip http://localhost:4502/etc/packages/tmp/repo/repo-oakindex-1519992982.zip
+ '[' 0 -ne 0 ']'
+ delete_pkg tmp/repo/repo-oakindex-1519992982.zip
+ pkg_mgr_http build '-X POST http://localhost:4502/crx/packmgr/service/.json/etc/packages/tmp/repo/repo-oakindex-1519992982.zip?cmd=delete'
++ curl_http '-X POST http://localhost:4502/crx/packmgr/service/.json/etc/packages/tmp/repo/repo-oakindex-1519992982.zip?cmd=delete'
++ curl -u admin:admin -f -s -S -X POST 'http://localhost:4502/crx/packmgr/service/.json/etc/packages/tmp/repo/repo-oakindex-1519992982.zip?cmd=delete'
+ out='{"success":true,"msg":"Package deleted"}'
+ [[ {"success":true,"msg":"Package deleted"} != *\"success\":true* ]]
+ pushd /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs
+ unzip -q pkg.zip
+ popd
+ [[ get == \d\i\f\f ]]
+ [[ get == \s\t\a\t\u\s ]]
+ '[' get == get ']'
+ true
+ unzip -l /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/pkg.zip
+ grep '   jcr_root/oak:index'
+ false
+ prompt 'download and overwrite locally?'
+ read -p 'download and overwrite locally? (y/n): ' -n 1 -r
download and overwrite locally? (y/n): y+ echo

+ [[ ! y =~ ^[Yy]$ ]]
+ rsync -avq --delete --exclude-from=/var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/.excludes /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/jcr_root//oak:index /Users/kitarek/repo-oak-index/jcr_root
rsync: link_stat "/var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs/jcr_root//oak:index" failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-52/rsync/main.c(996) [sender=2.6.9]
+ true
+ git rev-parse --git-dir
+ true
+ [[ -n '' ]]
+ [[ -n '' ]]
+ cleanup
+ '[' -d /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs ']'
+ rm -rf /var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.4xhFPNrs
+ exit

What's in the filesystem

> find . -type f 
> find . -type d
.
./jcr_root
./jcr_root/oak:index

Versions

rsync

rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

curl

curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy

[alexkli]

Have you tried repo checkout /_oak_index? This might work right now, although agree that this is a bit unfortunate for the checkout command, where you probably expect to pass the JCR path.

Generally, repo commands expect the file system path as argument, not the JCR path. Currently the mapping of the vault mapped paths for namespaces in file or directory names is only done in one direction, from mapped to jcr, such as _oak_index to oak:index.

[kitarek]

Thanks @alexkli for a workaround.
Yeah it works: repo checkout '/_oak_index'

+ local base=/var/folders/lg/v9sb730x7hx5x9nlnmsc8cx00000gn/T/repo.XXX.7klmsNQs
++ filesystem_to_jcr /_oak_index
++ local filter=/_oak_index
++ filter=/_oak_index
++ filter=/_oak_index
++ filter=/_oak_index
++ filter=/_oak_index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
++ filter=/oak:index
+++ urldecode /oak:index
+++ printf %b /oak:index
++ filter=/oak:index
++ echo /oak:index
+ local filter=/oak:index
+ local pkgGroup=tmp/repo
+ local pkgName=repo-oakindex
+ local pkgVersion=1520508470

OK I see now.

Probably users passing arguments to get command will expect the same:

get                download server content to local file system

Can we expect JCR and map it back to filesystem object? Is this really needed? I guess for JCR VLT filter you'll already have a correct value from command line (it needs to be sanitized if it's not) but you probably use rsync and you know what physically is in the package contents (archive)? An I right?

I'm thinking about pushing a PR for that in a free time.. but I need to be sure I won't break here any tool assumptions...

[alexkli]

Ok, thanks for verifying my hunch.

Actually, in the docs I already state that checkout gets a <jcr-path> as argument (raw JCR path), as opposed to <path> (file system path) as for all the others. So this should probably be fixed ;-) and we should have an inverse of the filesystem_to_jcr() function used for checkout.

This has to do essentially the same as jackrabbit-filevault's getPlatformName, replace some well known namespace prefixes (as a quick hack; if someone can write the real logic in bash that would be great), then escape the few special chars using %xx.

For all the other commands the local path must exist, so it can only be a local filevault mapped file system path. Nothing to do there I hope.

[synox]

I faced the same issue today. I would be great if checkout could handle the transformation. Or not transform at all, as macos does not mind the colons. Or update the docs to reflect this.