Adding shell utilities to Blink #281

holzschu · 2017-06-13T16:45:52Z

I suggest an extension to Blink, giving basic shell commands inside the shell (cp, rm, mv...)
This might help user in managing their private keys, or storing files locally.

This is based on GNU coreutils, edited so they compile for iOS and work as functions instead of commands. For each command (e.g. "ls") I rename its main function (e.g. "ls_main"), and call this function from inside MCPSession.m

The full list of commands available:
cd ln ls cp mv rm mkdir rmdir touch pwd
setenv env printenv
id groups realpath uname

For more functionalities, I suggest also compiling VimIOS and creating an App Group, so the two apps can share files. Once this is done, you can edit files in VimIOS and copy/delete them in Blink.

Only a small set of the GNU coreutils are available: some of them don't make sense in a sandboxed environment (kill, nohup, nice, chown), others don't work in a function because of local variables (df), still others send their output to STDOUT_FILENO instead of stdout, which was too complicated to change (cat).

To compile: 1) run ./configure 2) replace lib/config.h with lib/myConfig.h 3) open Xcode project 4) when compiled, move framework to ../Frameworks

To compile, use XCode project

To compile: use XCode project.

carloscabanero · 2017-06-13T17:57:33Z

Just a quick note before reviewing or talking in more detail. The GNU Coreutils are under the GPL3, and unfortunately it is incompatible with Blink - Mosh iOS clause. But maybe we can build the BSD versions?

holzschu · 2017-06-13T18:24:43Z

Hi,

On 13 Jun 2017, at 19:57, carloscabanero ***@***.***> wrote: Just a quick note before reviewing or talking in more detail. The GNU Coreutils are under the GPL3, and unfortunately it is incompatible with Blink - Mosh iOS clause. But maybe we can build the BSD versions?

Damn! I knew there would be some licensing issues, but not so fast. I'll look at the BSD version of the tools. On the plus side, the commands are easier to tinker with. Best, Nicolas Holzschuch

…

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#281 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHsmUHArvINDexm7TyyGFW2UCrGlTtX7ks5sDs2WgaJpZM4N4w3c>.

holzschu · 2017-06-14T11:15:05Z

Hi,

On 13 Jun 2017, at 19:57, carloscabanero ***@***.*** ***@***.***>> wrote: Just a quick note before reviewing or talking in more detail. The GNU Coreutils are under the GPL3, and unfortunately it is incompatible with Blink - Mosh iOS clause. But maybe we can build the BSD versions?

Quick question: is the *Apple* license OK? By far the easiest utilities to port are the ones from Apple's own Open Source effort: https://opensource.apple.com <https://opensource.apple.com/> Reading the source code, it seems to be following the BSD licence: /* * Copyright (c) 1989, 1993, 1994 * The Regents of the University of California. All rights reserved. * * This code is derived from software contributed to Berkeley by * Michael Fischbein. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ Best regards, Nicolas Holzschuch

carloscabanero · 2017-06-14T14:10:29Z

This is a tricky one, that is the original 4-clause BSD, which is incompatible with GPLv3. The problem is clause 3. But FreeBSD relicensed all their code to a compatible 3-clause one, including the Core Utils we care about (reason why in their license files they skip the 3). The code is technically the same, do you think we can use that one?

…ource initiative

openSource

carloscabanero · 2017-06-14T15:45:14Z

Thanks! I will prepare a new version for the Raw branch with this included. Don’t know what people will do with it, but interesting to figure out! :)

file_cmds is now complete (all commands that make sense / can work are implemented).

goerz · 2017-06-15T16:11:18Z

What would make this massively useful is to have vim included. I was trying to compile this myself, but I don't think I can, without a paid dev account (because of iCloud etc)

holzschu · 2017-06-15T20:27:17Z

I'll look into it, but it's a long shot.
It could work, but I don't think it's going to be easy. Vim needs a terminal that implements libcurses, which is a bit too much for Blink. The alternative would be to persuade the Vim developer to have a joint app group with Blink.

holzschu · 2018-04-27T08:01:24Z

Hi @yury,
the latest version of ios_system has wildcard expansion (ls *.txt, rm *.log). It is quite useful. You probably want that.

yury · 2018-04-27T08:04:12Z

Hi @holzschu,

Sure! that would be great. It will be v2.1 release? Or it is better to fork ios_system? What do you think?

yury · 2018-04-27T08:06:16Z

@holzschu, I also pass err stream to ios_system. That way curl | grep produce a little better output on the screen

holzschu · 2018-04-27T08:18:50Z

Hi @yury,

wildcard expansion might actually be already in 2.0. Does ls *.txt work?
I would prefer not to have a fork (unless it's useful), but ios_system as a submodule would definitely work.
And definitely yes about err stream.

yury · 2018-04-27T08:52:51Z

Hi @holzschu,

Yep, expansion working, so we are very good :)
I'm also not want to fork.
Ok.

carloscabanero · 2018-04-27T12:46:16Z

One more thing @holzschu, the last Blink version works with descriptors instead of streams, so we don't need to do fprintf all the time, etc... This may be useful and easier to port some of the tools and keep them in sync.

holzschu · 2018-04-27T12:53:52Z

I don't see the change to descriptors yet in the main branch. The switch from streams to descriptors will require some changes in ios_system and its API. With the current architecture, it should be relatively easy (but not instantaneous).

carloscabanero · 2018-04-27T22:09:00Z

Blink will continue to support both,. Just mentioned that in case it helps you with maintenance as usually this apps expect descriptors and not streams.

yury · 2018-04-28T11:28:21Z

Hi @holzschu,

We plan to migrate to new folders structure:

~/history.txt -> ~/.blink_history # or ~/.blink/history.txt
~/known_hosts -> ~/.ssh/known_hosts
~/defaults -> ~/.blink/defaults
~/hosts -> ~/.blink/hosts
~/keys -> ~/.blink/keys # or ~/.ssh/<keys>

Quick search showed that you do checks if files exists and fallback to known defaults in curl_ios_static/src/tool_operate.c

But maybe I missed something?

holzschu · 2018-04-28T11:56:04Z

Hi @yury,
Thanks for the heads up.
For known_hosts, yes, I search for it in ~/Documents/known_hosts and then in ~/Documents/.ssh/known_hosts, so the change should not affect ios_system.

I don't think I'm accessing the other ones.

yury · 2018-04-29T15:15:27Z

Ok, bad news...

Workaround on setlocale actually masks the problem. It doesn't crash, but mosh session resets its utf-8 locale somehow.

Scenario for replicating is more complex now.

first tab - mosh to server. Run htop or type any non ascii chars. - every thing is ok.
open second tab. Type ls -la.
switch to first tab. Run htop - will get bad screen. all non ascii chars will be replaced with <?> (On latest rn branch. On TF htop will behave strange)

Update:

I think that is our problem:
ls.c

Update:
Good news!
Yep, that was our problem.

Update:
ls -la loose color after I commented setlocale in it.

holzschu · 2018-04-29T20:16:59Z

Hi @yury,
I remember that setlocale depends on files that are outside the sandbox, and so it often returns NULL. Calling it has an impact, but checking the return value might be a bad idea. What happens if you comment lines 99 to 110 in mosh/src/util/locale_util.cc? (it might still crash in iosclient.cc, but why?)

set locale(LC_ALL,""), which is what everyone is calling, is supposed to get the locale value from the environment. But we didn't set it.

holzschu · 2018-04-30T07:11:21Z

Hi @yury,
I've been testing things .
set locale(LC_ALL, "") anywhere except in libmosh returns "C". Which is not the right answer (the terminal has UTF-8 abilities), but at least it doesn't crash.
set locale(LC_ALL, "") inside libmosh returns NULL. The question is: why only inside libmosh? Is there anything different with the terminal?

According to the man page, setlocale tries to access /usr/share/locale/locale/category and /usr/local/share/locale/locale/category, which don't exist in iOS sandboxed environment. One possibility would be to define $PATH_LOCALE and provide locale descriptions in $PATH_LOCALE/locale/category.

yury · 2018-04-30T07:21:12Z

hi @holzschu,

About commenting setlocale in mosh. We did that at first. It didn't crash, but utf8 sequences are corrupted.

About set PATH_LOCALE - yep, PATH_LOCALE is set in mosh ios client. What I did to prevent crash, I also set PATH_LOCALE in app delegate start. It stops crashing, but ls -la changes it concurent scenario (with tabs).

holzschu · 2018-04-30T07:55:47Z

Hi @yury,
I'm curious: what do you set PATH_LOCALE to?

yury · 2018-04-30T08:00:38Z

Hi @holzschu,

https://github.com/yury/blink/blob/rn/Blink/AppDelegate.m#L59

  NSString *locales_path = [mainBundle pathForResource:@"locales" ofType:@"bundle"];
  setenv("PATH_LOCALE", locales_path.UTF8String, forceOverwrite);

yury · 2018-04-30T08:28:32Z

yep, it is only LC_CTYPE in the bundle

holzschu · 2018-04-30T08:28:43Z

Hi @yury,
OK, thanks.
I got the latest version, plus a version of ios_system where ls.c calls set locale(). It seems to work (I don't see any differences on ls -la, for example). For more consistency, you could setenv LC_CTYPE to UTF-8 in AppDelegate.m, so every command uses UTF-8.

yury · 2018-04-30T08:46:06Z

Hi @holzschu,

Here the short video what I get.

holzschu · 2018-04-30T08:59:56Z

Hi @yury,
I'm going to install htop to reproduce. Right now, I have files named with emojis in the first terminal (running mosh) and ls -la in the second terminal.

holzschu · 2018-04-30T09:00:54Z

Quick question: does adding setenv("LC_CTYPE", "UTF-8", 1); on line 61 of AppDelegate.m change anything?

yury · 2018-04-30T09:05:53Z

actually not.

yury · 2018-04-30T09:08:30Z

oh, sorry!

It works!

yury · 2018-04-30T09:13:05Z

@holzschu thanks a lot! Testing...

carloscabanero · 2018-05-31T14:32:49Z

Alright, it has been quite a ride but we could finally put all the pieces together!!

Thanks a lot @holzschu for such a great work. We will continue contributing to tools on your repo 👍

holzschu · 2018-05-31T15:30:11Z

That is both awesome and overwhelming. It's awesome that you guys continue contributing to the tools!

N-Holzschuch added 9 commits June 13, 2017 11:16

Changes to Blink project so it can call shell utils (ls, cp...)

dfc8c49

Added coreutils-8.27 for source

513592c

To compile: 1) run ./configure 2) replace lib/config.h with lib/myConfig.h 3) open Xcode project 4) when compiled, move framework to ../Frameworks

Modified files so coreutils compile. Do not run configure.

14de6b6

Removing old version of coreutils-8.27

867f765

coreutils-8.27, modified to be a library and compile for iOS.

b876292

To compile, use XCode project

Parts of coreutil-8.27 package.

33f48d5

To compile: use XCode project.

more files, most of them not necessary:

5e031e5

Xcode project for blink

341fe2e

Added instructions for compilation.

4753e6c

N-Holzschuch added 3 commits June 14, 2017 17:34

Replacing source code for utilities from GNU coreutils to Apple openS…

de0fdd8

…ource initiative

Replacing the source for utilities from GNU coreutils to Apple

01a8bc1

openSource

Edited README to reflect new commands added and compilation instructions

032b7ab

N-Holzschuch added 5 commits June 14, 2017 17:47

Trying to remove coreutils

694fdc0

Edited README for mistakes

d2e96fe

Edited versions of the commands so they work.

26059d9

Added new commands

69f115f

Added new commands.

5d34052

file_cmds is now complete (all commands that make sense / can work are implemented).

N-Holzschuch added 6 commits June 16, 2017 22:45

Second library, shell_cmds, with shell based commands (env, pwd, ...)

1b6a28c

Modified source files so they compile on iOS.

81d6fc8

Edited Session to reflect the new commands

27f5e83

Modified README to reflect the new commands available

08b2fbb

Fixing some issues with df and du.

4baaf4f

One new framework, with grep, cat, wc.

9f04a53

yury added a commit to yury/blink that referenced this pull request Apr 27, 2018

Use err stream. Refs blinksh#281

35e48f8

yury added a commit to yury/blink that referenced this pull request Apr 30, 2018

Set env LC_CTYPE to UTF-8. refs blinksh#281

0a5d6ef

Merge branch 'master' of https://github.com/blinksh/blink

408782e

carloscabanero closed this May 31, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding shell utilities to Blink #281

Adding shell utilities to Blink #281

holzschu commented Jun 13, 2017

carloscabanero commented Jun 13, 2017

holzschu commented Jun 13, 2017 via email

holzschu commented Jun 14, 2017 via email

carloscabanero commented Jun 14, 2017

carloscabanero commented Jun 14, 2017

goerz commented Jun 15, 2017

holzschu commented Jun 15, 2017

holzschu commented Apr 27, 2018

yury commented Apr 27, 2018

yury commented Apr 27, 2018

holzschu commented Apr 27, 2018

yury commented Apr 27, 2018

carloscabanero commented Apr 27, 2018

holzschu commented Apr 27, 2018

carloscabanero commented Apr 27, 2018

yury commented Apr 28, 2018

holzschu commented Apr 28, 2018

yury commented Apr 29, 2018 •

edited

holzschu commented Apr 29, 2018

holzschu commented Apr 30, 2018

yury commented Apr 30, 2018

holzschu commented Apr 30, 2018

yury commented Apr 30, 2018

yury commented Apr 30, 2018

holzschu commented Apr 30, 2018

yury commented Apr 30, 2018

holzschu commented Apr 30, 2018

holzschu commented Apr 30, 2018

yury commented Apr 30, 2018

yury commented Apr 30, 2018

yury commented Apr 30, 2018

carloscabanero commented May 31, 2018

holzschu commented May 31, 2018

Adding shell utilities to Blink #281

Adding shell utilities to Blink #281

Conversation

holzschu commented Jun 13, 2017

carloscabanero commented Jun 13, 2017

holzschu commented Jun 13, 2017 via email

holzschu commented Jun 14, 2017 via email

carloscabanero commented Jun 14, 2017

carloscabanero commented Jun 14, 2017

goerz commented Jun 15, 2017

holzschu commented Jun 15, 2017

holzschu commented Apr 27, 2018

yury commented Apr 27, 2018

yury commented Apr 27, 2018

holzschu commented Apr 27, 2018

yury commented Apr 27, 2018

carloscabanero commented Apr 27, 2018

holzschu commented Apr 27, 2018

carloscabanero commented Apr 27, 2018

yury commented Apr 28, 2018

holzschu commented Apr 28, 2018

yury commented Apr 29, 2018 • edited

holzschu commented Apr 29, 2018

holzschu commented Apr 30, 2018

yury commented Apr 30, 2018

holzschu commented Apr 30, 2018

yury commented Apr 30, 2018

yury commented Apr 30, 2018

holzschu commented Apr 30, 2018

yury commented Apr 30, 2018

holzschu commented Apr 30, 2018

holzschu commented Apr 30, 2018

yury commented Apr 30, 2018

yury commented Apr 30, 2018

yury commented Apr 30, 2018

carloscabanero commented May 31, 2018

holzschu commented May 31, 2018

yury commented Apr 29, 2018 •

edited