Add game testing. #2050

o01eg · 2018-03-12T19:30:49Z

This PR adds tests for the game itself.
Test acts as a host for the game server. It starts singleplayer game, make a turns and check if server and AIs acts correctly and within time limits.
Test accepts environment variables FO_TEST_GAME_AIS to manage number of AI players, FO_TEST_GAME_TURNS to manage numbers of turns to make, and FO_TEST_GAME_SAVE to save game after each turn.
Test will stop when test player will be eliminated or someone wins the game.

Other changes:

Added Process::Terminate method to ensure server was stopped correctly.
ClientApp::VerifyCheckSum returns bool instead void.
Add server flag --testing to output AI logs to stdout.

geoffthemedio · 2018-03-14T13:49:38Z

test/game/TestGame.cpp

+                }
+                to_process = true;
+            } else {
+                if(to_process) {


Space between ( and if

o01eg · 2018-03-15T21:19:08Z

Rebased. Refactored and set time limits on game starting and turn processing.

o01eg · 2018-03-24T14:36:45Z

Rebased. Added environment variables TEST_GAME_AIS to manage number of AI players and TEST_GAME_TURNS to manage numbers of turns to make.

o01eg · 2018-04-10T07:39:26Z

Rebased. Implement testing saves. Add options TEST_GAME_SAVE to manage if saves will be applied (default true).

o01eg · 2018-05-03T20:35:17Z

Rebased. Tested against new docker image.

geoffthemedio · 2018-05-05T10:17:08Z

default/stringtables/en.txt

@@ -1186,6 +1186,9 @@ Start server in hostless mode. Server accepts players in the multiplayer lobby,
 OPTIONS_DB_SKIP_CHECKSUM
 Skip comparing the resource directory checksums.  This allows faster startup when client and server are on the same machine, using the same resource directory.

+OPTIONS_DB_TESTING
+Start server in testing mode. AI playing will be output log to colnsole instead file (Linux only).


colnsole
rewrite second sentence

Not clear if this option is Linux-only, or the to-consolve vs. to-file part is Linux-only

Option is no-op in other than Linux OS.

Put the "Linux only" at the start.

And perhaps don't even add it as an option in those cases...

Moved the option to macro condition for FREEORION_LINUX.

geoffthemedio · 2018-05-05T10:17:59Z

server/ServerApp.cpp

@@ -260,6 +260,14 @@ void ServerApp::CreateAIClients(const std::vector<PlayerSetupData>& player_setup
        args.push_back(GetOptionsDB().Get<std::string>("log-level"));
    }

+    if (GetOptionsDB().Get<bool>("testing")) {
+#ifdef FREEORION_LINUX
+        // Dirty hack to output log to console.


why not use the standard log files?

I'm not sure if it possible to get them from Travis CI to get detailed information about possible errors.

Running the tests isn't possible on AppVeyor as well?

It possible but logs will be output to files.

geoffthemedio · 2018-05-05T10:18:44Z

test/game/ClientAppFixture.cpp

+#include <boost/format.hpp>
+
+ClientAppFixture::ClientAppFixture()
+    :m_game_started(false)


: to previous line, with space between it and )

geoffthemedio · 2018-05-05T10:20:03Z

test/game/ClientAppFixture.cpp

+    // Dirty hack to output log to console.
+    InitLoggingSystem("/proc/self/fd/1", "Test");
+#else
+    InitLoggingSystem((GetUserDataDir() / "test.log").string(), "Test");


why not use standard log file location?

GetUserDataDir() is a standard folder for logs. test.log is used instead freeorion.log to distinguish unit test from human client.

geoffthemedio · 2018-05-05T10:20:30Z

test/game/ClientAppFixture.cpp

+}
+
+int ClientAppFixture::EffectsProcessingThreads() const
+{ return 1; }


Why not use standard option for this?

It requires unittest to have a fully initialized OptionsDB.

geoffthemedio · 2018-05-05T10:20:45Z

test/game/ClientAppFixture.cpp

+{ return m_networking->ConnectToLocalHostServer(); }
+
+void ClientAppFixture::HostSPGame(unsigned int num_AIs) {
+    std::vector<std::pair<std::string, std::string>> game_rules = GetGameRules().GetRulesAsStrings();


prefer auto ?

Stop test on elimination or win.

o01eg · 2018-06-19T04:37:06Z

Rebased.

o01eg · 2018-06-19T07:50:28Z

What if merge it into 0.4.8 branch too so release will be tested too?

geoffthemedio · 2018-06-19T08:22:07Z

no. adding new / potentially problematic stuff immediately before release not appropriate.

Vezzra · 2018-06-24T10:13:30Z

Considering the amount of changes to the codebase, this isn't a purely "infrastructure" PR anyway.

I'm going to add "component:internal" and a proper milestone (post release).

o01eg force-pushed the test-server branch 5 times, most recently from 6f03327 to a13b0ae Compare March 14, 2018 10:21

geoffthemedio added category:feature The Issue/PR describes or implements a new game feature. component:infrastructure The Issue/PR deals with development aiding infrastructure like CI, distribution or helper tools. labels Mar 14, 2018

geoffthemedio reviewed Mar 14, 2018

View reviewed changes

o01eg mentioned this pull request Mar 14, 2018

Segfault on second session in hostless mode #1897

Closed

o01eg force-pushed the test-server branch from 5e3e263 to 2dbb56a Compare March 15, 2018 20:17

o01eg force-pushed the test-server branch from 8b7f656 to d47373a Compare March 24, 2018 14:32

o01eg force-pushed the test-server branch from d47373a to b3b3f44 Compare April 10, 2018 07:33

o01eg mentioned this pull request Apr 28, 2018

Upgrade docker image to ubuntu:18.04 freeorion/freeorion-sdk#37

Closed

o01eg force-pushed the test-server branch 7 times, most recently from 2a098b0 to 07ecea5 Compare May 3, 2018 19:41

geoffthemedio reviewed May 5, 2018

View reviewed changes

o01eg added 18 commits June 18, 2018 23:54

Get server's binary path.

3805ff6

Lanunch and connect to server.

6fa8daf

Process messages.

519cb36

Try to diagnose Travis CI error.

c33eb86

Add Process::Terminate method to clean stop server.

d8300a5

Ignore SIGCHILD and system errors. Allow to kill server.

3b32ab6

Show failure status.

c15bb32

Successfully start game (on my PC).

c3c6cf4

Wait AI for turn.

9f47e3a

Make turn.

1b9e53c

Add testing mode to output AI log to console.

d06f339

Refactoring. Move Fixture to separate file.

9050192

Implement timeouts for waiting messages.

6ae5736

Add environment variables to manage number of turns and AIs.

b845205

Stop test on elimination or win.

Test saves. Add TEST_GAME_SAVE evironment option to disable saving.

ad62933

Grooming.

8d60238

Make testing option Linux-only.

d379bbf

Change it from unit test to smoke and system test.

1e72575

geoffthemedio merged commit ed13d4b into freeorion:master Jun 19, 2018

geoffthemedio added the status:merged All relevant commits of this PR were merged into the master development branch. label Jun 19, 2018

o01eg deleted the test-server branch June 19, 2018 08:52

o01eg mentioned this pull request Jun 19, 2018

Add system test for hostless mode. #2171

Merged

Vezzra added this to the post 0.4.8 milestone Jun 24, 2018

Vezzra added the component:internal The Issue/PR deals with any project component that has no explicit `component` label. label Jun 24, 2018

Vezzra modified the milestones: Next Release, v0.4.9 Sep 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add game testing. #2050

Add game testing. #2050

o01eg commented Mar 12, 2018 •

edited

Loading

geoffthemedio Mar 14, 2018

o01eg Mar 15, 2018

o01eg commented Mar 15, 2018

o01eg commented Mar 24, 2018

o01eg commented Apr 10, 2018

o01eg commented May 3, 2018

geoffthemedio May 5, 2018 •

edited

Loading

o01eg May 5, 2018

geoffthemedio May 5, 2018

o01eg May 5, 2018

geoffthemedio May 5, 2018 •

edited

Loading

o01eg May 5, 2018

geoffthemedio May 5, 2018

o01eg May 5, 2018

geoffthemedio May 5, 2018

o01eg May 5, 2018

geoffthemedio May 5, 2018

o01eg May 5, 2018

geoffthemedio May 5, 2018

o01eg May 5, 2018

geoffthemedio May 5, 2018

o01eg May 5, 2018

o01eg commented Jun 19, 2018

o01eg commented Jun 19, 2018

geoffthemedio commented Jun 19, 2018 •

edited

Loading

Vezzra commented Jun 24, 2018

Add game testing. #2050

Add game testing. #2050

Conversation

o01eg commented Mar 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

o01eg commented Mar 15, 2018

o01eg commented Mar 24, 2018

o01eg commented Apr 10, 2018

o01eg commented May 3, 2018

geoffthemedio May 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

geoffthemedio May 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

o01eg commented Jun 19, 2018

o01eg commented Jun 19, 2018

geoffthemedio commented Jun 19, 2018 • edited Loading

Vezzra commented Jun 24, 2018

o01eg commented Mar 12, 2018 •

edited

Loading

geoffthemedio May 5, 2018 •

edited

Loading

geoffthemedio May 5, 2018 •

edited

Loading

geoffthemedio commented Jun 19, 2018 •

edited

Loading