New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding config option to set superdataset root #2835

Merged
merged 9 commits into from Sep 19, 2018

Conversation

Projects
None yet
5 participants
@vsoch
Contributor

vsoch commented Sep 15, 2018

This pull request will close #2792 by adding a configuration value, datasets.locations.local_central_path

image

Notes

  • naming I would have called it superdataset_root or similar, but to be consistent I named it equivalently to the current variable LOCAL_CENTRAL_PATH --> datasets.locations.local_central_path.
  • The path defined in confs.py might still be useful for testing. I noticed that this default is set in the confs.py, and then used in testing. Since testing may be a different use case / invocation pathway than when it's actually being used, I left it here and didn't change the imports in testing.
  • docs my life was very easy! The change appeared there automatically :)

Changes

  • This change is complete

Please have a look @datalad/developers and thank you in advance for help with fine tuning. I'm still unfamiliar with the code base and using datalad.

vsoch added some commits Sep 15, 2018

@vsoch

This comment has been minimized.

Contributor

vsoch commented Sep 15, 2018

The error for the CI was that the datalad/metadata/search.py module was expected by the testing to have an attribute LOCAL_CENTRAL_PATH. Since this was a use case for testing, I re-imported it (as before) still maintaining that the primary functions in search.py use the one from cfg.

@vsoch

This comment has been minimized.

Contributor

vsoch commented Sep 16, 2018

hey @yarikoptic how should we deal with this test? With the change the default now goes to travis home:

  File "<string>", line 3, in raise_from
AssertionError: Expected call: install('/tmp/datalad_temp_test_search_outside1_install_central_dsr6s0t3sz', source='///')
Actual call: install('/home/travis/datalad', source='///')

so we would expect to change the test to fit. But on the other hand, instead of defaulting to home we might want to set the variable to be that still defined in the original configuration file? Let me know how you would like to address. I'm not sure why it seems to be passing in some of the grid?

@yarikoptic

This comment has been minimized.

Member

yarikoptic commented Sep 16, 2018

We have patch_config to patch config during tests. Pushed that fix + complete obliteration of the const variable... although now thinking what if someone uses it, so may be we should bend a bit to return it (might be tricky to avoid circular imports there)

@yarikoptic

This comment has been minimized.

Member

yarikoptic commented Sep 17, 2018

Pushed one more commit b7be9ba renaming local_central_path to central-dataset:

  • configuration variables cannot have underscores (git config thing)
  • previous variable name had to mention something about path, here we have already a section locations so decided to shorten it a bit. Since could be any dataset, didn't use "super" prefix
@yarikoptic

This comment has been minimized.

Member

yarikoptic commented Sep 17, 2018

@mih @kyleam @bpoldrack any feedback on the naming of the variable to be used?

@kyleam

This comment has been minimized.

Member

kyleam commented Sep 17, 2018

any feedback on the naming of the variable to be used?

I'm fine with "central-dataset". It's also refered to in the docs as the "canonical dataset":

``///``
refers to the "canonical" dataset located under `$HOME/datalad/`.
So running ``datalad install -d/// crcns`` will install the ``crcns`` subdataset
under ``$HOME/datalad/crcns``. This is the same as running
``datalad install $HOME/datalad/crcns``.

It might be worth reducing the number of ways we refer to it by either using "canonical-dataset" as the config variable name or updating the docs to use "central".

Also, this PR should update the hardcoded "$HOME/datalad/"s in that entry.

Since could be any dataset, didn't use "super" prefix

OK, though I'm not sure how much clarity that brings since the title and description in common_cfg.py say "superdataset".

@yarikoptic

This comment has been minimized.

Member

yarikoptic commented Sep 18, 2018

+1 to unify. I start to like central over canonical also since it is shorter. So let's adjust the docs (unless someone objects). doc should mention that it is a dataset, although typically a superdataset

@bpoldrack

This comment has been minimized.

Member

bpoldrack commented Sep 18, 2018

tbh, I think at some point we need a whole new name for this, since central, canonical, super - none of those is nailing it and might be confusing at some point. There's nothing special about the dataset itself and in theory you can use any dataset for that (not just the one from datalad.org).
Furthermore, I can imagine us extending the concept to configure even several of those standard "mountpoints" and "abbreviations", so you can configure /// to address the one at ~/datalad and an arbitrary number of similar associations of a symbol and a location.

However, central is just as good as any other for now and unifying the mentions of it is way more important than an actual discussion about naming and future changes, I guess. So, basically I'm fine with that. ;-)

@vsoch

This comment has been minimized.

Contributor

vsoch commented Sep 18, 2018

You don't think something along the lines of "root" is appropriate? It mirrors the ROOT location well, for example for Singularity the root (where we assume / to be when we build the container) is SINGULARITY_ROOTFS. It doesn't say "this is a particular kind of container (or dataset) but rather "this is where I'm rooting the command for now"

@bpoldrack

This comment has been minimized.

Member

bpoldrack commented Sep 18, 2018

I kind of like root, but still - it somewhat suggests, that you are required to install everything beneath it or something along this line, which is misleading. You can have datasets with no connection to this "root" location whatsoever.

but rather "this is where I'm rooting the command for now"but rather "this is where I'm rooting the command for now"

This notion gets us closer indeed, I think. We would need to say something like:
/// is our "root" symbol and it refers to the dataset you configured to serve as this root.

Yeah, may be root works. :-)

@yarikoptic

This comment has been minimized.

Member

yarikoptic commented Sep 18, 2018

TBH, not sure about root, which has a connotation of the common point from which everything "grows". Here we are talking only about some sample/canonical/central dataset which we install and use in search if currently we are not within any other dataset. If I setup some root I expect current operation to operate on that location, which wouldn't be the case if we are within some other dataset.

@vsoch

This comment has been minimized.

Contributor

vsoch commented Sep 18, 2018

I don't think root is implied to be a single, nonchanging point. With chroot it's for a process and it can be anywhere on the file system. Understanding it as the root where everything grows from, even if it's just temporary, I think does describe it fairly well. And without this PR, I can't even change this, which is problematic. In the same way that $GOHOME is used as a root, or even $HOME I think that most users would like have (at least locally) a common point to interact with their datasets, and if that changes for a one off action they know they can use a flag to designate (akin to Singularity --home flag).

@yarikoptic

This comment has been minimized.

Member

yarikoptic commented Sep 18, 2018

ok, to proceed forward then let's just keep central-dataset. I will tune up/push docs now to use more uniform wording

@yarikoptic

This comment has been minimized.

Member

yarikoptic commented Sep 18, 2018

or actually -- what about default-dataset? that is the default dataset we install/search after all.

@yarikoptic yarikoptic force-pushed the vsoch:add/superdataset-root branch from b7be9ba to a6b3282 Sep 18, 2018

@yarikoptic

This comment has been minimized.

Member

yarikoptic commented Sep 19, 2018

If no objections or comments I would like to merge it today

@vsoch

This comment has been minimized.

Contributor

vsoch commented Sep 19, 2018

No objections from this peanut gallery! Is the CI error something that can be overlooked?

@vsoch

This comment has been minimized.

Contributor

vsoch commented Sep 19, 2018

Just one in the matrix, for Python 3.5

 Raw log
 step_start_instance
Starting instance
✓ selected image "travis-ci-garnet-trusty-1512502259-986baf0"
✓ rendered startup script
✓ inserted instance
• sleeping 5s before checking instance insert
• polling for instance insert completion...
✓ instance is ready (7.514s)
step_upload_script
Uploading script
• waiting for ssh connectivity.............
✓ ssh connectivity established (20.589s)
✓ uploaded script
worker_info
Worker information
hostname: 4c3ddda9-168b-4e79-b74d-4fc2bcc56d7a@1.production-2-worker-org-gce-8ntg
version: v4.1.2 https://github.com/travis-ci/worker/tree/91246b057ccd93649046771f29221f839c19a7d3
instance: travis-job-6efc6061-ad7d-47f3-87bf-b0c0cc224943 travis-ci-garnet-trusty-1512502259-986baf0 (via amqp)
startup: 7.514229946s
system_info
Build system information
Build language: python
Build group: stable
Build dist: trusty
Build id: 430174024
Job id: 430174048
Runtime kernel version: 4.4.0-101-generic
travis-build version: 0bb83ee18
Build image provisioning date and time
Tue Dec  5 19:58:13 UTC 2017
Operating System Details
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.5 LTS
Release:	14.04
Codename:	trusty
Cookbooks Version
7c2c6a6 https://github.com/travis-ci/travis-cookbooks/tree/7c2c6a6
git version
git version 2.15.1
bash version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
gcc version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:38 2017
 OS/Arch:      linux/amd64
Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:20 2017
 OS/Arch:      linux/amd64
 Experimental: false
clang version
clang version 5.0.0 (tags/RELEASE_500/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/clang-5.0.0/bin
jq version
jq-1.5
bats version
Bats 0.4.0
shellcheck version
0.4.6
shfmt version
v2.0.0
ccache version
ccache version 3.1.9
Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2011 Joel Rosdahl
This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; either version 3 of the License, or (at your option) any later
version.
cmake version
cmake version 3.9.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
heroku version
heroku-cli/6.14.39-addc925 (linux-x64) node-v9.2.0
imagemagick version
Version: ImageMagick 6.7.7-10 2017-07-31 Q16 http://www.imagemagick.org
md5deep version
4.2
mercurial version
Mercurial Distributed SCM (version 4.2.2)
(see https://mercurial-scm.org for more information)
Copyright (C) 2005-2017 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
mysql version
mysql  Ver 14.14 Distrib 5.6.33, for debian-linux-gnu (x86_64) using  EditLine wrapper
openssl version
OpenSSL 1.0.1f 6 Jan 2014
packer version
Packer v1.0.2
Your version of Packer is out of date! The latest version
is 1.1.2. You can update by downloading from www.packer.io
postgresql client version
psql (PostgreSQL) 9.6.6
ragel version
Ragel State Machine Compiler version 6.8 Feb 2013
Copyright (c) 2001-2009 by Adrian Thurston
subversion version
svn, version 1.8.8 (r1568071)
   compiled Aug 10 2017, 17:20:39 on x86_64-pc-linux-gnu
Copyright (C) 2013 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/
The following repository access (RA) modules are available:
* ra_svn : Module for accessing a repository using the svn network protocol.
  - with Cyrus SASL authentication
  - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
  - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using serf.
  - using serf 1.3.3
  - handles 'http' scheme
  - handles 'https' scheme
sudo version
Sudo version 1.8.9p5
Configure options: --prefix=/usr -v --with-all-insults --with-pam --with-fqdn --with-logging=syslog --with-logfac=authpriv --with-env-editor --with-editor=/usr/bin/editor --with-timeout=15 --with-password-timeout=0 --with-passprompt=[sudo] password for %p:  --without-lecture --with-tty-tickets --disable-root-mailer --enable-admin-flag --with-sendmail=/usr/sbin/sendmail --with-timedir=/var/lib/sudo --mandir=/usr/share/man --libexecdir=/usr/lib/sudo --with-sssd --with-sssd-lib=/usr/lib/x86_64-linux-gnu --with-selinux
Sudoers policy plugin version 1.8.9p5
Sudoers file grammar version 43
Sudoers path: /etc/sudoers
Authentication methods: 'pam'
Syslog facility if syslog is being used for logging: authpriv
Syslog priority to use when user authenticates successfully: notice
Syslog priority to use when user authenticates unsuccessfully: alert
Send mail if the user is not in sudoers
Use a separate timestamp for each user/tty combo
Lecture user the first time they run sudo
Root may run sudo
Allow some information gathering to give useful error messages
Require fully-qualified hostnames in the sudoers file
Visudo will honor the EDITOR environment variable
Set the LOGNAME and USER environment variables
Length at which to wrap log file lines (0 for no wrap): 80
Authentication timestamp timeout: 15.0 minutes
Password prompt timeout: 0.0 minutes
Number of tries to enter a password: 3
Umask to use or 0777 to use user's: 022
Path to mail program: /usr/sbin/sendmail
Flags for mail program: -t
Address to send mail to: root
Subject line for mail messages: *** SECURITY information for %h ***
Incorrect password message: Sorry, try again.
Path to authentication timestamp dir: /var/lib/sudo
Default password prompt: [sudo] password for %p:
Default user to run commands as: root
Value to override user's $PATH with: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Path to the editor for use by visudo: /usr/bin/editor
When to require a password for 'list' pseudocommand: any
When to require a password for 'verify' pseudocommand: all
File descriptors >= 3 will be closed before executing a command
Environment variables to check for sanity:
	TZ
	TERM
	LINGUAS
	LC_*
	LANGUAGE
	LANG
	COLORTERM
Environment variables to remove:
	RUBYOPT
	RUBYLIB
	PYTHONUSERBASE
	PYTHONINSPECT
	PYTHONPATH
	PYTHONHOME
	TMPPREFIX
	ZDOTDIR
	READNULLCMD
	NULLCMD
	FPATH
	PERL5DB
	PERL5OPT
	PERL5LIB
	PERLLIB
	PERLIO_DEBUG
	JAVA_TOOL_OPTIONS
	SHELLOPTS
	GLOBIGNORE
	PS4
	BASH_ENV
	ENV
	TERMCAP
	TERMPATH
	TERMINFO_DIRS
	TERMINFO
	_RLD*
	LD_*
	PATH_LOCALE
	NLSPATH
	HOSTALIASES
	RES_OPTIONS
	LOCALDOMAIN
	CDPATH
	IFS
Environment variables to preserve:
	JAVA_HOME
	TRAVIS
	CI
	DEBIAN_FRONTEND
	XAUTHORIZATION
	XAUTHORITY
	PS2
	PS1
	PATH
	LS_COLORS
	KRB5CCNAME
	HOSTNAME
	HOME
	DISPLAY
	COLORS
Locale to use while parsing sudoers: C
Directory in which to store input/output logs: /var/log/sudo-io
File in which to store the input/output log: %{seq}
Add an entry to the utmp/utmpx file when allocating a pty
PAM service name to use
PAM service name to use for login shells
Create a new PAM session for the command to run in
Maximum I/O log sequence number: 0
Local IP address and netmask pairs:
	10.240.0.28/255.255.255.255
	172.17.0.1/255.255.0.0
Sudoers I/O plugin version 1.8.9p5
gzip version
gzip 1.6
Copyright (C) 2007, 2010, 2011 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.
Written by Jean-loup Gailly.
zip version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.
Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip
as of above date; see http://www.info-zip.org/ for other sites.
Compiled with gcc 4.8.2 for Unix (Linux ELF) on Oct 21 2013.
Zip special compilation options:
	USE_EF_UT_TIME       (store Universal Time)
	BZIP2_SUPPORT        (bzip2 library version 1.0.6, 6-Sept-2010)
	    bzip2 code and library copyright (c) Julian R Seward
	    (See the bzip2 license for terms of use)
	SYMLINK_SUPPORT      (symbolic links supported)
	LARGE_FILE_SUPPORT   (can read and write large files on file system)
	ZIP64_SUPPORT        (use Zip64 to store large files in archives)
	UNICODE_SUPPORT      (store and read UTF-8 Unicode paths)
	STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field)
	UIDGID_NOT_16BIT     (old Unix 16-bit UID/GID extra field not used)
	[encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3)
Encryption notice:
	The encryption code of this program is not copyrighted and is
	put in the public domain.  It was originally written in Europe
	and, to the best of our knowledge, can be freely distributed
	in both source and object forms from any country, including
	the USA under License Exception TSU of the U.S. Export
	Administration Regulations (section 740.13(e)) of 6 June 2002.
Zip environment options:
             ZIP:  [none]
          ZIPOPT:  [none]
vim version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Nov 24 2016 16:43:18)
Included patches: 1-52
Extra patches: 8.0.0056
Modified by pkg-vim-maintainers@lists.alioth.debian.org
Compiled by buildd@
Huge version without GUI.  Features included (+) or not (-):
+acl             +farsi           +mouse_netterm   +syntax
+arabic          +file_in_path    +mouse_sgr       +tag_binary
+autocmd         +find_in_path    -mouse_sysmouse  +tag_old_static
-balloon_eval    +float           +mouse_urxvt     -tag_any_white
-browse          +folding         +mouse_xterm     -tcl
++builtin_terms  -footer          +multi_byte      +terminfo
+byte_offset     +fork()          +multi_lang      +termresponse
+cindent         +gettext         -mzscheme        +textobjects
-clientserver    -hangul_input    +netbeans_intg   +title
-clipboard       +iconv           +path_extra      -toolbar
+cmdline_compl   +insert_expand   -perl            +user_commands
+cmdline_hist    +jumplist        +persistent_undo +vertsplit
+cmdline_info    +keymap          +postscript      +virtualedit
+comments        +langmap         +printer         +visual
+conceal         +libcall         +profile         +visualextra
+cryptv          +linebreak       +python          +viminfo
+cscope          +lispindent      -python3         +vreplace
+cursorbind      +listcmds        +quickfix        +wildignore
+cursorshape     +localmap        +reltime         +wildmenu
+dialog_con      -lua             +rightleft       +windows
+diff            +menu            -ruby            +writebackup
+digraphs        +mksession       +scrollbind      -X11
-dnd             +modify_fname    +signs           -xfontset
-ebcdic          +mouse           +smartindent     -xim
+emacs_tags      -mouseshape      -sniff           -xsmp
+eval            +mouse_dec       +startuptime     -xterm_clipboard
+ex_extra        +mouse_gpm       +statusline      -xterm_save
+extra_search    -mouse_jsbterm   -sun_workshop    -xpm
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H     -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc   -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed -o vim        -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl    -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
iptables version
iptables v1.4.21
curl version
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
wget version
GNU Wget 1.15 built on linux-gnu.
rsync version
rsync  version 3.1.0  protocol version 31
gimme version
v1.2.0
nvm version
0.33.6
perlbrew version
/home/travis/perl5/perlbrew/bin/perlbrew  - App::perlbrew/0.80
phpenv version
rbenv 1.1.1-25-g6aa70b6
rvm version
rvm 1.29.3 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
default ruby version
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
CouchDB version
couchdb 1.6.1
ElasticSearch version
5.5.0
Installed Firefox version
firefox 56.0.2
MongoDB version
MongoDB 3.4.10
PhantomJS version
2.1.1
Pre-installed PostgreSQL versions
9.2.24
9.3.20
9.4.15
9.5.10
9.6.6
RabbitMQ Version
3.6.14
Redis version
redis-server 4.0.6
riak version
2.2.3
Pre-installed Go versions
1.7.4
ant version
Apache Ant(TM) version 1.9.3 compiled on April 8 2014
mvn version
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
Maven home: /usr/local/maven-3.5.2
Java version: 1.8.0_151, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-98-generic", arch: "amd64", family: "unix"
gradle version
------------------------------------------------------------
Gradle 4.0.1
------------------------------------------------------------
Build time:   2017-07-07 14:02:41 UTC
Revision:     38e5dc0f772daecca1d2681885d3d85414eb6826
Groovy:       2.4.11
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.8.0_151 (Oracle Corporation 25.151-b12)
OS:           Linux 4.4.0-98-generic amd64
lein version
Leiningen 2.8.1 on Java 1.8.0_151 Java HotSpot(TM) 64-Bit Server VM
Pre-installed Node.js versions
v4.8.6
v6.12.0
v6.12.1
v8.9
v8.9.1
phpenv versions
  system
  5.6
* 5.6.32 (set by /home/travis/.phpenv/version)
  7.0
  7.0.25
  7.1
  7.1.11
  hhvm
  hhvm-stable
composer --version
Composer version 1.5.2 2017-09-11 16:59:25
Pre-installed Ruby versions
ruby-2.2.7
ruby-2.3.4
ruby-2.4.1
Network availability confirmed.
Setting APT mirror in /etc/apt/sources.list: http://us-central1.gce.archive.ubuntu.com/ubuntu/
3.5 is not installed; attempting download
Downloading archive: https://s3.amazonaws.com/travis-python-archives/binaries/ubuntu/14.04/x86_64/python-3.5.tar.bz2
$ curl -sSf -o python-3.5.tar.bz2 ${archive_url}
curl: (56) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
Unable to download 3.5 archive. The archive may not exist. Please consider a different version.
@kyleam

This comment has been minimized.

Member

kyleam commented Sep 19, 2018

If no objections or comments I would like to merge it today

I'm fine with any of the names proposed. I still find the docs confusing.

``///``
refers to the "default" dataset located under `$HOME/datalad/`.
So running ``datalad install -d/// crcns`` will install the ``crcns`` subdataset
under ``$HOME/datalad/crcns``. This is the same as running
``datalad install $HOME/datalad/crcns``.

This requires that the user has read the "Downloading DataLad's default superdataset" section that mentions datalad.locations.default-dataset, remembered it, and understood it well enough that they realize $HOME/datalad/ is just the default.

@codecov

This comment has been minimized.

codecov bot commented Sep 19, 2018

Codecov Report

Merging #2835 into master will not change coverage.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #2835   +/-   ##
=======================================
  Coverage   90.21%   90.21%           
=======================================
  Files         246      246           
  Lines       31789    31789           
=======================================
  Hits        28677    28677           
  Misses       3112     3112
Impacted Files Coverage Δ
datalad/support/network.py 85.97% <ø> (ø) ⬆️
datalad/consts.py 100% <ø> (ø) ⬆️
datalad/metadata/tests/test_search.py 93.23% <100%> (+0.05%) ⬆️
datalad/distribution/dataset.py 94% <100%> (ø) ⬆️
datalad/metadata/search.py 79.42% <100%> (ø) ⬆️
datalad/distribution/tests/test_dataset.py 100% <100%> (ø) ⬆️
datalad/interface/common_cfg.py 100% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17150e6...a6b3282. Read the comment docs.

@mih

This comment has been minimized.

Member

mih commented Sep 19, 2018

I am fine with any name change, or none at all. I personally do not like the special casing of ///. We did not implement any of the semantics that were envisioned back then, but instead this is just a symbol for a particular dataset. Any flavor of "super", "root", "central" comes with the chance that people interpret this dataset as special, or worse: required. Any such interpretation would be wrong, and the docs misleading. However, given that we do special-case /// I see no way to avoid this.

@yarikoptic

This comment has been minimized.

Member

yarikoptic commented Sep 19, 2018

@vsoch - I do not see any failure in travis atm, may be it was restarted --looked like a fluke. note that we have a bunch of 3.5 runs, so do not even know which exactly was it

@kyleam - I will merge, feel welcome to adjust the wording in a follow up PR

@mih - so it seems that you like "default" more even without saying it, so I will proceed.

@yarikoptic yarikoptic merged commit 477fd81 into datalad:master Sep 19, 2018

5 checks passed

WIP ready for review
Details
codecov/patch 100% of diff hit (target 90.21%)
Details
codecov/project 90.21% (+0%) compared to 17150e6
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@vsoch vsoch deleted the vsoch:add/superdataset-root branch Sep 20, 2018

@vsoch

This comment has been minimized.

Contributor

vsoch commented Sep 20, 2018

Thanks again @yarikoptic !

@yarikoptic yarikoptic added this to the Release 0.10.4 milestone Oct 22, 2018

yarikoptic added a commit that referenced this pull request Oct 24, 2018

Merge tag '0.11.0' into debian
 Upgrade of [git-annex] to the most recent available to your release is
 advisable since a number of issues were resolved at that level.

 ### Major refactoring and deprecations

 - `datalad.consts.LOCAL_CENTRAL_PATH` constant was deprecated in favor
   of `datalad.locations.default-dataset` [configuration] variable
   ([#2835])

 ### Minor refactoring

 - `"notneeded"` messages are no longer reported by default results
   renderer
 - [run] no longer shows commit instructions upon command failure when
   `explicit` is true and no outputs are specified ([#2922])
 - `get_git_dir` moved into GitRepo ([#2886])
 - `_gitpy_custom_call` removed from GitRepo ([#2894])
 - `GitRepo.get_merge_base` argument is now called `commitishes` instead
   of `treeishes` ([#2903])

 ### Fixes

 - [update] should not leave the dataset in non-clean state ([#2858])
   and some other enhancements ([#2859])
 - Fixed chunking of the long command lines to account for decorators
   and other arguments ([#2864])
 - Progress bar should not crash the process on some missing progress
   information ([#2891])
 - Default value for `jobs` set to be `"auto"` (not `None`) to take
   advantage of possible parallel get if in `-g` mode ([#2861])
 - [wtf] must not crash if `git-annex` is not installed etc ([#2865]),
   ([#2865]), ([#2918]), ([#2917])
 - Fixed paths (with spaces etc) handling while reporting annex error
   output ([#2892]), ([#2893])
 - `__del__` should not access `.repo` but `._repo` to avoid attempts
   for reinstantiation etc ([#2901])
 - Fix up submodule `.git` right in `GitRepo.add_submodule` to avoid
   added submodules being non git-annex friendly ([#2909]), ([#2904])
 - [run-procedure] ([#2905])
   - now will provide dataset into the procedure if called within dataset
   - will not crash if procedure is an executable without `.py` or `.sh`
     suffixes
 - Use centralized `.gitattributes` handling while setting annex backend
   ([#2912])
 - `GlobbedPaths.expand(..., full=True)` incorrectly returned relative
    paths when called more than once ([#2921])

 ### Enhancements and new features

 - Report progress on [clone] when installing from "smart" git servers
   ([#2876])
 - Stale/unused `sth_like_file_has_content` was removed ([#2860])
 - Enhancements to [search] to operate on "improved" metadata layouts
   ([#2878])
 - Output of `git annex init` operation is now logged ([#2881])
 - New
   - `GitRepo.cherry_pick` ([#2900])
   - `GitRepo.format_commit` ([#2902])
 - [run-procedure] ([#2905])
   - procedures can now recursively be discovered in subdatasets as well.
     The uppermost has highest priority
   - Procedures in user and system locations now take precedence over
     those in datasets.

* tag '0.11.0':
  Make it a 0.11.0 release since there were some API RFings
  REL: slight tune up to Changelog following the advices
  DOC: v0.10.4: Mention change in procedure precedence (a0cbcba)
  DOC: v0.10.4: Fix description of db715b7
  DOC: v0.10.4: Improve description of 6f615a4
  DOC: v0.10.4: Remove duplicate word

yarikoptic added a commit that referenced this pull request Oct 24, 2018

Merge tag '0.11.0' into debian
 ## 0.11.0 (Oct 23, 2018) -- Soon-to-be-perfect

 [git-annex] 6.20180913 (or later) is now required - provides a number of
 fixes for v6 mode operations etc.

 ### Major refactoring and deprecations

 - `datalad.consts.LOCAL_CENTRAL_PATH` constant was deprecated in favor
   of `datalad.locations.default-dataset` [configuration] variable
   ([#2835])

 ### Minor refactoring

 - `"notneeded"` messages are no longer reported by default results
   renderer
 - [run] no longer shows commit instructions upon command failure when
   `explicit` is true and no outputs are specified ([#2922])
 - `get_git_dir` moved into GitRepo ([#2886])
 - `_gitpy_custom_call` removed from GitRepo ([#2894])
 - `GitRepo.get_merge_base` argument is now called `commitishes` instead
   of `treeishes` ([#2903])

 ### Fixes

 - [update] should not leave the dataset in non-clean state ([#2858])
   and some other enhancements ([#2859])
 - Fixed chunking of the long command lines to account for decorators
   and other arguments ([#2864])
 - Progress bar should not crash the process on some missing progress
   information ([#2891])
 - Default value for `jobs` set to be `"auto"` (not `None`) to take
   advantage of possible parallel get if in `-g` mode ([#2861])
 - [wtf] must not crash if `git-annex` is not installed etc ([#2865]),
   ([#2865]), ([#2918]), ([#2917])
 - Fixed paths (with spaces etc) handling while reporting annex error
   output ([#2892]), ([#2893])
 - `__del__` should not access `.repo` but `._repo` to avoid attempts
   for reinstantiation etc ([#2901])
 - Fix up submodule `.git` right in `GitRepo.add_submodule` to avoid
   added submodules being non git-annex friendly ([#2909]), ([#2904])
 - [run-procedure] ([#2905])
   - now will provide dataset into the procedure if called within dataset
   - will not crash if procedure is an executable without `.py` or `.sh`
     suffixes
 - Use centralized `.gitattributes` handling while setting annex backend
   ([#2912])
 - `GlobbedPaths.expand(..., full=True)` incorrectly returned relative
    paths when called more than once ([#2921])

 ### Enhancements and new features

 - Report progress on [clone] when installing from "smart" git servers
   ([#2876])
 - Stale/unused `sth_like_file_has_content` was removed ([#2860])
 - Enhancements to [search] to operate on "improved" metadata layouts
   ([#2878])
 - Output of `git annex init` operation is now logged ([#2881])
 - New
   - `GitRepo.cherry_pick` ([#2900])
   - `GitRepo.format_commit` ([#2902])
 - [run-procedure] ([#2905])
   - procedures can now recursively be discovered in subdatasets as well.
     The uppermost has highest priority
   - Procedures in user and system locations now take precedence over
     those in datasets.

* tag '0.11.0':
  CHANGELOG adjusted to reflect new minimal version of git-annex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment