New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UTF-8 (rather, any non-ASCII) evidently dropped/ignored #26

Open
kriskowal opened this Issue Nov 13, 2017 · 2 comments

Comments

Projects
None yet
2 participants
@kriskowal
Copy link

kriskowal commented Nov 13, 2017

Steps to reproduce:

Parse, for example, πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦

Expected: constituent UTF-8 bytes of the four unicode characters, and the three unicode zero-width join characters, should all be send to Print.

Actual: handler.Print is not called. No other method appears to be called.

package main

import ansiterm "github.com/Azure/go-ansiterm"

func main() {
	parser := ansiterm.CreateParser("Ground", &handler{})
	parser.Parse([]byte("πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦"))
}

type handler struct{}

func (h *handler) Print(b byte) error {
	panic("not implemented")
}

func (h *handler) Execute(b byte) error {
	panic("not implemented")
}

func (h *handler) CUU(int) error {
	panic("not implemented")
}

func (h *handler) CUD(int) error {
	panic("not implemented")
}

func (h *handler) CUF(int) error {
	panic("not implemented")
}

func (h *handler) CUB(int) error {
	panic("not implemented")
}

func (h *handler) CNL(int) error {
	panic("not implemented")
}

func (h *handler) CPL(int) error {
	panic("not implemented")
}

func (h *handler) CHA(int) error {
	panic("not implemented")
}

func (h *handler) VPA(int) error {
	panic("not implemented")
}

func (h *handler) CUP(int, int) error {
	panic("not implemented")
}

func (h *handler) HVP(int, int) error {
	panic("not implemented")
}

func (h *handler) DECTCEM(bool) error {
	panic("not implemented")
}

func (h *handler) DECOM(bool) error {
	panic("not implemented")
}

func (h *handler) DECCOLM(bool) error {
	panic("not implemented")
}

func (h *handler) ED(int) error {
	panic("not implemented")
}

func (h *handler) EL(int) error {
	panic("not implemented")
}

func (h *handler) IL(int) error {
	panic("not implemented")
}

func (h *handler) DL(int) error {
	panic("not implemented")
}

func (h *handler) ICH(int) error {
	panic("not implemented")
}

func (h *handler) DCH(int) error {
	panic("not implemented")
}

func (h *handler) SGR([]int) error {
	panic("not implemented")
}

func (h *handler) SU(int) error {
	panic("not implemented")
}

func (h *handler) SD(int) error {
	panic("not implemented")
}

func (h *handler) DA([]string) error {
	panic("not implemented")
}

func (h *handler) DECSTBM(int, int) error {
	panic("not implemented")
}

func (h *handler) IND() error {
	panic("not implemented")
}

func (h *handler) RI() error {
	panic("not implemented")
}

func (h *handler) Flush() error {
	return nil
}
@ncw

This comment has been minimized.

Copy link

ncw commented Sep 5, 2018

I can confirm this problem. It happens for a single unicode character such as Β£ too.

@ncw

This comment has been minimized.

Copy link

ncw commented Sep 11, 2018

Note also that some chinese characters cause an error to be reported by the parser "New state of 'nil' is invalid", eg replacing the above main with this one

func main() {
	parser := ansiterm.CreateParser("Ground", &handler{})
	n, err := parser.Parse([]byte("η‹ε¦–ε°ηΊ’ε¨˜"))
	log.Print(n, err)
}

Gives

$ go run ansiterm.go 
2018/09/11 21:29:53 3 New state of 'nil' is invalid.

ncw added a commit to ncw/rclone that referenced this issue Oct 6, 2018

cmd: Don't print non-ASCII characters with --progress on windows - fixes
 #2501

This bug causes lots of strange behaviour with non-ASCII characters and --progress

Azure/go-ansiterm#26

ncw added a commit to ncw/rclone that referenced this issue Oct 10, 2018

cmd: Don't print non-ASCII characters with --progress on windows - fixes
 #2501

This bug causes lots of strange behaviour with non-ASCII characters and --progress

Azure/go-ansiterm#26

ncw added a commit to ncw/rclone that referenced this issue Oct 11, 2018

cmd: Don't print non-ASCII characters with --progress on windows - fixes
 #2501

This bug causes lots of strange behaviour with non-ASCII characters and --progress

Azure/go-ansiterm#26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment