Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fs ls command for dbfs #429

Merged
merged 17 commits into from
Jun 5, 2023
Merged

Add fs ls command for dbfs #429

merged 17 commits into from
Jun 5, 2023

Conversation

shreyas-goenka
Copy link
Contributor

@shreyas-goenka shreyas-goenka commented Jun 2, 2023

Changes

  1. Adds fs ls command
  2. Adds ability to define multiple templates

Tests

Manually and integration tests

@shreyas-goenka
Copy link
Contributor Author

shreyas.goenka@THW32HFW6T bricks % cli fs ls dbfs:/FileStore -l --output=json
[
  {
    "name":"babynames.csv",
    "is_directory":false,
    "size":278154,
    "last_modified":"2023-03-05T16:27:11+01:00"
  },
  {
    "name":"jars",
    "is_directory":true,
    "size":0,
    "last_modified":"2022-10-03T08:18:02+02:00"
  },
  {
    "name":"tables",
    "is_directory":true,
    "size":0,
    "last_modified":"2022-06-23T15:45:57+02:00"
  },
  {
    "name":"terraform-test",
    "is_directory":true,
    "size":0,
    "last_modified":"2023-03-26T20:18:06+02:00"
  }
]%
shreyas.goenka@THW32HFW6T bricks % cli fs ls dbfs:/FileStore -l
FILE      278154 2023-03-05T15:27:11Z babynames.csv
DIRECTORY 0 2022-10-03T06:18:02Z jars
DIRECTORY 0 2022-06-23T13:45:57Z tables
DIRECTORY 0 2023-03-26T18:18:06Z terraform-test
shreyas.goenka@THW32HFW6T bricks % cli fs ls dbfs:/FileStore
babynames.csv
jars
tables
terraform-test

@shreyas-goenka
Copy link
Contributor Author

cmd/fs/ls.go Outdated Show resolved Hide resolved
cmd/fs/ls.go Outdated Show resolved Hide resolved
cmd/fs/ls.go Outdated Show resolved Hide resolved
ModTime time.Time `json:"last_modified"`
}

func toLsOutput(f fs.DirEntry) (*lsOutput, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would make it a method of fs.DirEntry instead, for better readability

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We cannot define methods on structs defined in external packages right?

cmd/fs/ls.go Outdated Show resolved Hide resolved
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change modifying cmdio to use a map of templates doesn't seem necessary. Please revert that and stick to calling RenderWithTemplate with an actual template.

libs/cmdio/render.go Outdated Show resolved Hide resolved
cmd/fs/ls.go Outdated Show resolved Hide resolved
libs/filer/dbfs_client.go Outdated Show resolved Hide resolved
cmd/fs/ls_output.go Outdated Show resolved Hide resolved
libs/cmdio/render.go Outdated Show resolved Hide resolved
cmd/fs/ls.go Outdated
{{end}}
`))
}
return cmdio.Render(ctx, jsonDirEntries)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please specify the template here as well. Now this command it taking two different approaches for rendering a template based on the mode which is weird to the uninitiated reader.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a significance for storing templates in the annotations in the first place?

cc: @nfx

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has to be done for autogenerated commands because assignment of the annotations field happens in a different file from where the command is defined. Means a stable way to associate metadata with the command instance.

cmd/fs/ls.go Outdated Show resolved Hide resolved
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, final comment.

Thanks!

@@ -0,0 +1,104 @@
package internal
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you name this file fs_ls_test.go to disambiguate from (potential) other ls tests?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the other potential ls tests possible? did you have anything specific in mind?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

workspace fs

@shreyas-goenka
Copy link
Contributor Author

@shreyas-goenka shreyas-goenka merged commit 6ff0012 into main Jun 5, 2023
4 checks passed
@shreyas-goenka shreyas-goenka deleted the fs-ls branch June 5, 2023 15:41
@pietern pietern mentioned this pull request Jun 12, 2023
pietern added a commit that referenced this pull request Jun 12, 2023
## Changes

CLI:
* Add directory tracking to sync
([#425](#425)).
* Add fs cat command for dbfs files
([#430](#430)).
* Add fs ls command for dbfs
([#429](#429)).
* Add fs mkdirs command for dbfs
([#432](#432)).
* Add fs rm command for dbfs
([#433](#433)).
* Add installation instructions
([#458](#458)).
* Add new line to cmdio JSON rendering
([#443](#443)).
* Add profile on `databricks auth login`
([#423](#423)).
* Add readable console logger
([#370](#370)).
* Add workspace export-dir command
([#449](#449)).
* Added secrets input prompt for secrets put-secret command
([#413](#413)).
* Added spinner when loading command prompts
([#420](#420)).
* Better error message if can not load prompts
([#437](#437)).
* Changed service template to correctly handle required positional
arguments ([#405](#405)).
* Do not generate prompts for certain commands
([#438](#438)).
* Do not prompt for List methods
([#411](#411)).
* Do not use FgWhite and FgBlack for terminal output
([#435](#435)).
* Skip path translation of job task for jobs with a Git source
([#404](#404)).
* Tweak profile prompt
([#454](#454)).
* Update with the latest Go SDK
([#457](#457)).
* Use cmdio in version command for `--output` flag
([#419](#419)).

Bundles:
* Check for nil environment before accessing it
([#453](#453)).

Dependencies:
* Bump github.com/hashicorp/terraform-json from 0.16.0 to 0.17.0
([#459](#459)).
* Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19
([#412](#412)).

Internal:
* Add Mkdir and ReadDir functions to filer.Filer interface
([#414](#414)).
* Add Stat function to filer.Filer interface
([#421](#421)).
* Add check for path is a directory in filer.ReadDir
([#426](#426)).
* Add fs.FS adapter for the filer interface
([#422](#422)).
* Add implementation of filer.Filer for local filesystem
([#460](#460)).
* Allow equivalence checking of filer errors to fs errors
([#416](#416)).
* Fix locker integration test
([#417](#417)).
* Implement DBFS filer
([#139](#139)).
* Include recursive deletion in filer interface
([#442](#442)).
* Make filer.Filer return fs.DirEntry from ReadDir
([#415](#415)).
* Speed up sync integration tests
([#428](#428)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants