Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Sync std{in,out,err} Encodings with `Terminal` #11234

Closed
wants to merge 3 commits into from

4 participants

@execjosh

This is a spin off from #11168.

This modification keeps the system.std{in,out,err} encodings in sync with the Terminal. That is, whenever the Terminal encoding gets changed, that update will now be propagated to the System module.

SIDE NOTE: Should the reverse case (propagate system.std{in,out,err} encoding change to Terminal) be handled?

execjosh added some commits
@execjosh execjosh Make `Terminal` emit a signal upon encoding change 8c721cf
@execjosh execjosh Teach `File` how to change its encoding
If a `File` is in "text" mode, then it has an encoding.  This
encoding defaults to UTF-8; however, it can be set only at time of
construction (by using `fs.open`).

This modification allows the user to change the encoding on-the-fly
for "text" mode `File` instances.
69808d1
@execjosh execjosh Synchronize std{in,out,err} encoding with `Terminal` a4ab6c2
@ariya
Owner

Do you want this to be merged as a squashed commit? If not, we still need to refer to the issue for later tracking.

@JamesMGreene
Collaborator

@execjosh: Regarding your side note:

Should the reverse case (propagate system.std{in,out,err} encoding change to Terminal) be handled?

Does the Terminal encoding need to match the system.std{in, out, err} encoding in order to read/write characters properly? If so, then I would definitely say "yes". If not, then I'm not certain... maybe still "yes" because you'd want that encoding setting to persist across different system.std{in, out, err} "sessions" within a single PhantomJS run but then you should also revert the Terminal's encoding to its original when PhantomJS exits.

@execjosh

@ariya: I wasn't really sure what "the issue" should be... I guess should I link to both #11168 and #11162 in all of the commits?

As for squashing, I don't really have any preference; so, whatever granularity you think is best for the history is probably best :smiley:

@execjosh

@JamesMGreene: Thank you for your input!

Terminal is a singleton that monopolizes c{out,err} (essentially any output from the C++ side, and, I guess, console.log). It is purely internal to PhantomJS and should have absolutely no effect on the actual shell environment running phantomjs. The encoding used by Terminal can be set via --output-encoding on the command line (it initially gets set to UTF-8).

system.std{in,out,err} are File instances that use UTF-8 for their encoding, each instance having their own, separate pointer to the text codec. If at some point in time the Terminal instance encoding gets changed, then they will not be updated, which could possibly lead to unexpected behavior. Hence this modification.

However, if the user changes one of system.std{out,err}'s encoding, I had thought that it might be a good idea to propagate back to Terminal. But, that could lead to system.stdout and Terminal being one encoding, and system.stderr being in another.

Hmm... It might actually be best just not to do any syncing in either direction at all, maybe? Hmm...

I guess the big problem is that Terminal, system.stdin, system.stdout, and system.stderr each have their own encodings...

@execjosh

Any opinions?

@ariya
Owner

Sorry for the late response, will have a look this evening.

@ariya ariya referenced this pull request from a commit
@execjosh execjosh Teach `File` how to change its encoding
If a `File` is in "text" mode, then it has an encoding.  This
encoding defaults to UTF-8; however, it can be set only at time of
construction (by using `fs.open`).

This modification allows the user to change the encoding on-the-fly
for "text" mode `File` instances.

See #11234 #11234
Spin off from #11168 #11168
b159144
@ariya
Owner

Merged. Thanks!

@ariya ariya closed this
@execjosh execjosh deleted the branch
@execjosh

If anyone's getting link errors after applying this, they may need to do touch src/phantomjs.pro (or the equivalent on their OS) so that qmake rebuilds the moc stuff for Terminal.

@Vitallium
Collaborator

Exactly! I forgot to re-run qmake :(

@JamesMGreene JamesMGreene referenced this pull request from a commit
@execjosh execjosh Implement system.std{in,out,err}
See [issue 333][1] and pull request #192.

**Caveat**

`File::read` currently takes no parameters and is equivalent to a
"`readAll`".  This will be changed later to match [IO/A Spec's
`Stream#read`][2]; but, should still be noted.

[1]: http://code.google.com/p/phantomjs/issues/detail?id=333
[2]: http://wiki.commonjs.org/wiki/IO/A#Instance_Methods
f6c8722
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 14, 2013
  1. @execjosh
  2. @execjosh

    Teach `File` how to change its encoding

    execjosh authored
    If a `File` is in "text" mode, then it has an encoding.  This
    encoding defaults to UTF-8; however, it can be set only at time of
    construction (by using `fs.open`).
    
    This modification allows the user to change the encoding on-the-fly
    for "text" mode `File` instances.
  3. @execjosh
This page is out of date. Refresh to see the latest.
View
4 src/encoding.h
@@ -46,11 +46,11 @@ class Encoding
QString getName() const;
void setEncoding(const QString &encoding);
+ QTextCodec *getCodec() const;
+
static const Encoding UTF8;
private:
- QTextCodec *getCodec() const;
-
QTextCodec *m_codec;
static const QByteArray DEFAULT_CODEC_NAME;
};
View
43 src/filesystem.cpp
@@ -216,6 +216,49 @@ void File::close()
deleteLater();
}
+bool File::setEncoding(const QString &encoding) {
+ if (encoding.isEmpty() || encoding.isNull()) {
+ return false;
+ }
+
+ // "Binary" mode doesn't use/need text codecs
+ if ((QTextStream *)NULL == m_fileStream) {
+ // TODO: Should we switch to "text" mode?
+ return false;
+ }
+
+ // Since there can be multiple names for the same codec (i.e., "utf8" and
+ // "utf-8"), we need to get the codec in the system first and use its
+ // canonical name
+ QTextCodec *codec = QTextCodec::codecForName(encoding.toAscii());
+ if ((QTextCodec *)NULL == codec) {
+ return false;
+ }
+
+ // Check whether encoding actually needs to be changed
+ const QString encodingBeforeUpdate(m_fileStream->codec()->name());
+ if (0 == encodingBeforeUpdate.compare(QString(codec->name()), Qt::CaseInsensitive)) {
+ return false;
+ }
+
+ m_fileStream->setCodec(codec);
+
+ // Return whether update was successful
+ const QString encodingAfterUpdate(m_fileStream->codec()->name());
+ return 0 != encodingBeforeUpdate.compare(encodingAfterUpdate, Qt::CaseInsensitive);
+}
+
+QString File::getEncoding() const
+{
+ QString encoding;
+
+ if ((QTextStream *)NULL != m_fileStream) {
+ encoding = QString(m_fileStream->codec()->name());
+ }
+
+ return encoding;
+}
+
// private:
bool File::_isUnbuffered() const
View
3  src/filesystem.h
@@ -64,6 +64,9 @@ public slots:
void flush();
void close();
+ QString getEncoding() const;
+ bool setEncoding(const QString &encoding);
+
private:
bool _isUnbuffered() const;
View
24 src/system.cpp
@@ -37,6 +37,7 @@
#include <QTextCodec>
#include "../env.h"
+#include "terminal.h"
System::System(QObject *parent) :
QObject(parent)
@@ -124,6 +125,8 @@ System::System(QObject *parent) :
m_os.insert("name", "unknown");
m_os.insert("version", "unknown");
#endif
+
+ connect(Terminal::instance(), SIGNAL(encodingChanged(QString)), this, SLOT(_onTerminalEncodingChanged(QString)));
}
System::~System()
@@ -203,10 +206,29 @@ QObject *System::_stdin() {
return m_stdin;
}
+// private slots:
+
+void System::_onTerminalEncodingChanged(const QString &encoding)
+{
+ if ((File *)NULL != m_stdin) {
+ m_stdin->setEncoding(encoding);
+ }
+
+ if ((File *)NULL != m_stdout) {
+ m_stdout->setEncoding(encoding);
+ }
+
+ if ((File *)NULL != m_stderr) {
+ m_stderr->setEncoding(encoding);
+ }
+}
+
// private:
File *System::createFileInstance(QFile *f)
{
- QTextCodec *codec = QTextCodec::codecForName("UTF-8");
+ // Get the Encoding used by the Terminal at this point in time
+ Encoding e(Terminal::instance()->getEncoding());
+ QTextCodec *codec = e.getCodec();
return new File(f, codec, this);
}
View
3  src/system.h
@@ -76,6 +76,9 @@ class System : public QObject
// system.stdin
QObject *_stdin();
+private slots:
+ void _onTerminalEncodingChanged(const QString &encoding);
+
private:
File *createFileInstance(QFile *f);
View
26 src/terminal.cpp
@@ -54,9 +54,33 @@ QString Terminal::getEncoding() const
return m_encoding.getName();
}
-void Terminal::setEncoding(const QString &encoding)
+bool Terminal::setEncoding(const QString &encoding)
{
+ // Since there can be multiple names for the same codec (i.e., "utf8" and
+ // "utf-8"), we need to get the codec in the system first and use its
+ // canonical name
+ QTextCodec *codec = QTextCodec::codecForName(encoding.toAscii());
+ if ((QTextCodec *)NULL == codec) {
+ return false;
+ }
+
+ // Check whether encoding actually needs to be changed
+ const QString encodingBeforeUpdate(m_encoding.getName());
+ if (0 == encodingBeforeUpdate.compare(QString(codec->name()), Qt::CaseInsensitive)) {
+ return false;
+ }
+
m_encoding.setEncoding(encoding);
+
+ // Emit the signal only if the encoding actually was changed
+ const QString encodingAfterUpdate(m_encoding.getName());
+ if (0 == encodingBeforeUpdate.compare(encodingAfterUpdate, Qt::CaseInsensitive)) {
+ return false;
+ }
+
+ emit encodingChanged(encoding);
+
+ return true;
}
void Terminal::cout(const QString &string, const bool newline) const
View
7 src/terminal.h
@@ -39,15 +39,20 @@
class Terminal: public QObject
{
+ Q_OBJECT
+
public:
static Terminal *instance();
QString getEncoding() const;
- void setEncoding(const QString &encoding);
+ bool setEncoding(const QString &encoding);
void cout(const QString &string, const bool newline = true) const;
void cerr(const QString &string, const bool newline = true) const;
+signals:
+ void encodingChanged(const QString &encoding);
+
private:
void output(std::ostream &out, const QString &string, const bool newline) const;
View
68 test/fs-spec-01.js
@@ -59,6 +59,74 @@ describe("Basic Files API (read, write, remove, ...)", function() {
expect(content).toEqual("hello\nworld\nasdf\n");
});
+ it("should be able to get the encoding (default: UTF-8)", function() {
+ var encoding = "";
+ try {
+ var f = fs.open(FILENAME, "r");
+ encoding = f.getEncoding();
+ f.close();
+ } catch (e) {
+ console.log(e);
+ }
+ expect(encoding).toEqual("UTF-8");
+ });
+
+ it("should be able to set the encoding via options", function() {
+ var encoding = "";
+ try {
+ var f = fs.open(FILENAME, {
+ charset: "UTF-8"
+ , mode: "r"
+ });
+ encoding = f.getEncoding();
+ f.close();
+ } catch (e) {
+ console.log(e);
+ }
+ expect(encoding).toEqual("UTF-8");
+
+ try {
+ var f = fs.open(FILENAME, {
+ charset: "SJIS"
+ , mode: "r"
+ });
+ encoding = f.getEncoding();
+ f.close();
+ } catch (e) {
+ console.log(e);
+ }
+ expect(encoding).toEqual("Shift_JIS");
+ });
+
+ it("should be able to change the encoding", function() {
+ var encoding = "";
+ try {
+ var f = fs.open(FILENAME, {
+ charset: "UTF-8"
+ , mode: "r"
+ });
+ f.setEncoding("utf8");
+ encoding = f.getEncoding();
+ f.close();
+ } catch (e) {
+ console.log(e);
+ }
+ expect(encoding).toEqual("UTF-8");
+
+ try {
+ var f = fs.open(FILENAME, {
+ charset: "SJIS"
+ , mode: "r"
+ });
+ f.setEncoding("eucjp");
+ encoding = f.getEncoding();
+ f.close();
+ } catch (e) {
+ console.log(e);
+ }
+ expect(encoding).toEqual("EUC-JP");
+ });
+
it("should be able to copy a file", function() {
expect(fs.exists(FILENAME_COPY)).toBeFalsy();
fs.copy(FILENAME, FILENAME_COPY);
Something went wrong with that request. Please try again.