Skip to content

Commit

Permalink
* Added simplified "ck init repo" to create a CK repository in the cu…
Browse files Browse the repository at this point in the history
…rrent directory

* Added docs about "ck init repo"
  • Loading branch information
gfursin committed Oct 20, 2020
1 parent fb5a026 commit 509efd7
Show file tree
Hide file tree
Showing 4 changed files with 138 additions and 11 deletions.
2 changes: 2 additions & 0 deletions CHANGES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
* Fixed issue when adding the "init" action to modules (this function always exists in CK modules).
We create "new_init" function in such case.
* Checking all function names in the CK module when adding a new action
* Added simplified "ck init repo" to create a CK repository in the current directory
* Added docs about "ck init repo"

* V1.15.0
* Improved the integration with the open CK portal: https://cKnowledge.io
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ in the form of reusable artifacts and portable workflows with a common API, CLI,
and meta description. See how CK supports collaborative and reproducible AI, ML, and Systems R&D:
* [Real-world use-cases](https://cKnowledge.org/partners.html) from Arm, General Motors, IBM, MLPerf, the Raspberry Pi foundation, and ACM
* [Reddit discussion about reproducing 150 papers](https://www.reddit.com/r/MachineLearning/comments/ioq8do/n_reproducing_150_research_papers_the_problems)
* [Project overview (preprint; to appear soon in the journal)](https://doi.org/10.6084/m9.figshare.12988361)
* [Project overview (preprint; to appear in the journal soon)](https://doi.org/10.6084/m9.figshare.12988361)

## News

Expand Down Expand Up @@ -111,7 +111,7 @@ any program (image corner detection in our case)
with any compatible data set on any compatible platform:

```bash
pip install ck
python3 -m pip install ck

ck pull repo --url=https://github.com/ctuning/ck-crowdtuning

Expand Down
121 changes: 112 additions & 9 deletions ck/repo/module/repo/module.py
Original file line number Diff line number Diff line change
Expand Up @@ -689,7 +689,9 @@ def update(i):
dn=x
changed=True

# If remote, update URL
# If remote or shared, update URL
if shared=='yes' or shared=='git': d['shared']='git'

shared=d.get('shared','')
if remote=='yes':
url=d.get('url','')
Expand All @@ -704,6 +706,8 @@ def update(i):
d['url']=x
changed=True
elif shared!='':
if url!='': d['url']=url

url=d.get('url','')
ck.out('Repository is shared ...')
ck.out('')
Expand Down Expand Up @@ -2818,6 +2822,11 @@ def cp(i):
def new_init(i):
"""
Input: {
(data_uoa) - repo name
(data_uid) - repo UID
(url) - URL if shared
(deps) - list of deps on other repositories separated by comma
(path) - path where to initialize a repository
}
Output: {
Expand All @@ -2828,15 +2837,109 @@ def new_init(i):
"""

ck.out('init CK repo in a given directory')
# Check if global writing is allowed
r=ck.check_writing({})
if r['return']>0: return r

ck.out('')
ck.out('Command line: ')
ck.out('')
# Check output
o=i.get('out','')
oo=''
if o=='con': oo=o

# Check path
path=os.getcwd()
if i.get('path','')!='':
path=i['path']
if not os.path.isdir(path):
return {'return':1, 'error':'path not found'}

# Check name
data_uoa=i.get('data_uoa','')
data_uid=i.get('data_uid','')

if data_uoa=='':
# Check from .ckr.json
pckr=os.path.join(path, ck.cfg['repo_file'])
if os.path.isfile(pckr):
r=ck.load_json_file({'json_file':pckr})
if r['return']>0: return r
d=r['dict']

import json
cmd=json.dumps(i, indent=2)
data_uoa=d.get('data_uoa','')
data_uid=d.get('data_uid','')

ck.out(cmd)
if data_uoa!='':
ck.out('Detected CK repo name: '+data_uoa)

return {'return':0}
if data_uoa=='':
# Try to detect from .ckr.json
r=ck.inp({'text':'Enter CK repository name: '})
if r['return']>0: return r
data_uoa=r['string'].strip()

# Check if already exists
exists=False
r=ck.access({'action':'load',
'module_uoa':work['self_module_uid'],
'data_uoa':data_uoa})
if r['return']>0 and r['return']!=16: return r
if r['return']==0:
exists=True

r=ck.inp({'text':'CK repository "'+data_uoa+'" already exists. Update (y/N)? '})
if r['return']>0: return r
s=r['string'].strip().lower()

if s=='' or s=='n' or s=='no':
return {'return':1, 'error':'CK repository already exists'}

# Check URL
url=i.get('url','')
if url=='':
# Attempt to check from current Git dir
pgit=os.path.join(path, '.git', 'config')
if os.path.isfile(pgit):
r=ck.load_text_file({'text_file':pgit, 'split_to_list':'yes'})
if r['return']>0: return r
ll=r['lst']

for l in ll:
l=l.strip()
if l.startswith('url ='):
url=l[5:].strip()
ck.out('Detected URL: '+url)
break

# prepare deps
repo_deps=[]

deps=i.get('deps','').strip().split(',')

for q in deps:
q=q.strip()
if q!='':
repo_deps.append({'repo_uoa':q})

# Create or update repo
ii={'module_uoa':work['self_module_uid'],
'data_uoa':data_uoa,
'quiet':'yes',
'path':path,
'out':oo}

if data_uid!='': ii['data_uid']=data_uid

if exists:
ii['action']='update'
else:
ii['action']='add'

if url!='':
ii['shared']='yes'
ii['url']=url

if len(repo_deps)>0:
ii['repo_deps']=repo_deps

r=ck.access(ii)
return r
22 changes: 22 additions & 0 deletions docs/src/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,28 @@ ck {action} ... @input.yaml
* Automation actions are implemented using the internal CK module [*repo*]( https://cknowledge.io/c/module/repo ).
* See the list of all automation actions and their API at [cKnowledge.io platform]( https://cknowledge.io/c/module/repo/#api ).

### Init new CK repository in the current path
```bash
ck init repo
```

CK will ask user for a repo name and will also attempt to detect Git URL from .git/config.

Extra options:

```bash
ck init repo:{CK repo name}
ck init repo --url={Git URL with the CK repository}
ck init repo --url={Git URL with the CK repository} --deps={list of CK repos}
```

Example:

```
ck init repo:asplos21-artifact123 --url=https://github.com/ctuning/ck-asplos21-artifact123 --deps=ck-autotuning
ck init repo:mlperf-submission --url=https://github.com/ctuning/ck-mlperf-submission321 --deps=ck-mlperf
```

### Pull existing repository using Git URL

Expand Down

0 comments on commit 509efd7

Please sign in to comment.