On the efficiency of back testing?

[hlevel]

Hello, may I ask you two questions? :)

1.How to improve the back test efficiency? tickers-xrp-usdt. There are 47073 pieces of data in the TSV file. It is estimated that you can run 8 pieces in one second, which takes a total of 100 minutes. How can you improve the efficiency?

2.BackTesting a policy will load all *.tsv files must be run and completed before the back test is completed. Can you change it to back test only one currency pairs?

My code:
`@Autowired
private TickerFluxMock tickerFluxMock;

/** Dumb strategy. */
@Autowired
private SimpleTa4jStrategy strategy;

@test
@DisplayName("Check gains")
public void gainTest() {
await().forever().until(() -> tickerFluxMock.isFluxDone());

final Map<CurrencyDTO, GainDTO> gains = strategy.getGains();

System.out.println("Cumulated gains:");
gains.forEach((currency, gain) -> System.out.println(currency + " : " + gain.getAmount()));

System.out.println("Position closed:");
strategy.getPositions()
        .values()
        .stream()
        .filter(p -> p.getStatus().equals(OrderStatusDTO.CLOSED))
        .forEach(p -> System.out.println(" - " + p.getDescription()));

System.out.println("Position not closed:");
strategy.getPositions()
        .values()
        .stream()
        .filter(p -> !p.getStatus().equals(OrderStatusDTO.CLOSED))
        .forEach(p -> System.out.println(" - " + p.getDescription()));

assertTrue(gains.get(strategy.getRequestedCurrencyPair().getQuoteCurrency()).getPercentage() > 0);

}`

My config:
test\java\resources
.....application.properties
.....tickers-btc-usdt.tsv
.....tickers-xrp-usdt.tsv
.....user-main.tsv
.....user-trade.tsv

[straumat]

Hi!

• Point 1: sadly, I cannot parallelize the process, each ticker must be treated completely (with orders and trades created if needed) before reading another one... I see no way to optimize it.

• Point 2 : Instead await().forever().until(() -> tickerFluxMock.isFluxDone());, you can do await().forever().until(() -> tickerFluxMock.isFluxDone(new CurrencyPairDTO(ETH, USDT)));.

[hlevel]

Hi, I've tried to use the following code, which seems to end quickly without running.

await().forever().until(() -> tickerFluxMock.isFluxDone(new CurrencyPairDTO(BTC, USDT)));

Console output:
Cumulated gains:
Position closed:
Position not closed:

[straumat]

@hlevel you have to adapt my code and change the CurrencyPairDTO to the one you want to use

[hlevel]

Hi, I don't understand what you mean?
I've tried the code solution you gave, and it didn't work, use version 5.0.6

This is my process

This source logic always seems to return true.

This method is successful.

[straumat]

@hlevel Is tickers-btc-usdt.tsv full ? can you attach the file ?
To debug things, can you add a log on onTickersUpdates() in your strategy to see if the tickers-btc-usdt.tsv file is really read ? thx

[hlevel]

I use the Maven prototype to generate the project, and the data is also generated. The onTickersUpdates method does not print any data
tickers_src.zip

[straumat]

@hlevel Indeed, there is an issue with my method tickerFluxMock.isFluxDone(CurrencyPairDTO) I will fix this as soon as I find where it comes from

[straumat]

I think the issue is fixed now, it will be available in 5.0.8-SNAPSHOT if this action executes correctly: https://github.com/cassandre-tech/cassandre-trading-bot/actions/runs/1679646906

[hlevel]

OK, in addition, p.getDescription() it is recommended to print the order information back, and then add the time information of when to place an order and when to sell. Without time information, it is difficult to compare the trend strategy chart!

[straumat]

@hlevel I think you should use p.getOpeningOrder() and get the field you want. Some people think getDescription() already returns too much information.

[hlevel]

@straumat Hi,Share a piece of code for loading back test data and generating files, which can be integrated into the function

Get data load file:

@Slf4j
public class CandleHistoryBackFill {
    private final static String CANDLES_URL = "https://api.kucoin.com/api/v1/market/candles?type=%s&symbol=%S&startAt=%S&endAt=%S";
    private final static String OUT_FOLDER = "src/test/resources";
    private final static String OUT_FILE = "tickers-%S-%S.tsv";
    private final static String OUT_SYMBOL = "%S-%S";
    private final int backfillHour = 24;

    private final RestTemplate restTemplate;
    private final ObjectMapper objectMapper = new ObjectMapper();

    private Candlestick candlestick = Candlestick.m1;
    private int backfillLength = 6;
    private ChronoUnit backfillUnit = ChronoUnit.HOURS;

    private LocalDateTime backfillFrom = null;
    private LocalDateTime backfillTo = null;

    CandleHistoryBackFill() {
        this(null, null);
    }

    CandleHistoryBackFill(String host, Integer port) {
        restTemplate = this.getRestTemplate(host, port);
    }

    public LocalDateTime backfillTo() {
        if (backfillTo != null) {
            return backfillTo;
        }
        return LocalDateTime.now();
    }

    public LocalDateTime backfillFrom() {
        if (backfillFrom != null) {
            return backfillFrom;
        }
        return backfillTo().minus(backfillLength, backfillUnit);
    }

    public void backfillFrom(LocalDateTime backfillFrom) {
        this.backfillFrom = backfillFrom;
    }

    public void backfillFrom(LocalDateTime backfillFrom, Candlestick candlestick) {
        this.backfillFrom = backfillFrom;
        this.candlestick = candlestick;
    }

    public final void backfillHistory(CurrencyPairDTO... currencyPairToUpdate) {
        LinkedHashSet<CurrencyPairDTO> allSymbols = new LinkedHashSet<>();
        Collections.addAll(allSymbols, currencyPairToUpdate);
        backfillHistory(allSymbols);
    }

    public final void backfillHistory(Collection<CurrencyPairDTO> currencyPairs) {
        final Instant start = this.backfillFrom().toInstant(ZoneOffset.of("+8"));
        final Instant end = this.backfillTo().toInstant(ZoneOffset.of("+8"));
        log.info("Created {} Candle file between {} and {}", currencyPairs.stream().map(CurrencyPairDTO::toString).collect(Collectors.joining(",")), DateTimeUtil.formatDateFromUnix(start.toEpochMilli(), DateTimeUtil.YYYY_MM_DD_HH_MM_SS), DateTimeUtil.formatDateFromUnix(end.toEpochMilli(), DateTimeUtil.YYYY_MM_DD_HH_MM_SS));

        List<String[]> retryGapList = new ArrayList<>();
        Iterator<CurrencyPairDTO> iterator = currencyPairs.stream().iterator();
        while (iterator.hasNext()) {
            CurrencyPairDTO currencyPair = iterator.next();
            File file = this.generateFile(currencyPair);
            if(file.exists()) {
                log.info("Current file {} already exists", file.getName());
                continue;
            }

            List<String[]> gapList = this.separateGapList(currencyPair, start, end);
            if(gapList.isEmpty()) {
                log.info("{} has no data to load", currencyPair);
                continue;
            }

            List<String[]> currencyPairCandleList = new ArrayList<>();
            for (String[] gap : gapList) {
                List<String[]> candleList = this.candleLoader(gap, 3);
                if(candleList.isEmpty()) {
                    retryGapList.add(gap);
                    continue;
                }
                currencyPairCandleList.addAll(candleList);
                retryGapList.addAll(this.analyticalCandleCaps(gap[0], candleList));
            }
            this.generateDataFile(currencyPairCandleList, file);
        }
    }

    private final List<String[]> separateGapList(CurrencyPairDTO currencyPair, Instant start, Instant end) {
        Date startDate = DateTimeUtil.parseDateFormUnix(start.toEpochMilli());
        Date endDate = DateTimeUtil.parseDateFormUnix(end.toEpochMilli());
        List<String[]> gapList = new ArrayList<>();
        while (DateTimeUtil.compareTowDays(startDate, endDate) < 1) {
            Date newDate = DateTimeUtil.addHours(startDate, backfillHour);
            String symbol = String.format(OUT_SYMBOL, currencyPair.getBaseCurrency(), currencyPair.getQuoteCurrency());
            long s = startDate.getTime()/1000;
            long e = newDate.getTime()/1000;
            gapList.add(new String[]{symbol, String.valueOf(s), String.valueOf(e)});
            startDate = newDate;
        }
        return gapList;
    }

    private List<String[]> candleLoader(String[] gap, Integer retry) {
        try{
            if(retry <= 0) {
                return Lists.newArrayList();
            }
            String candleUrl = String.format(CANDLES_URL, candlestick.code, gap[0], gap[1], gap[2]);
            /*
            String s = DateTimeUtil.formatDateFromUnix(Long.parseLong(gap[1])*1000, null);
            String e = DateTimeUtil.formatDateFromUnix(Long.parseLong(gap[2])*1000, null);
            System.out.println(s + "," + e + "," + candleUrl);
             */
            Thread.sleep(3000l);
            ResponseEntity<String> responseEntity = restTemplate.getForEntity(candleUrl, String.class);
            if(responseEntity.getStatusCode() != HttpStatus.OK) {
                return this.candleLoader(gap, retry - 1);
            }
            Map<String, Object> mapJSON =  objectMapper.readValue(responseEntity.getBody(), new TypeReference<>(){});
            if(mapJSON.containsKey("code")) {
                int code = Integer.parseInt(String.valueOf(mapJSON.get("code")));
                switch (code) {
                    case 200000:
                        List<String[]> candleList = objectMapper.readValue(String.valueOf(mapJSON.get("data")), new TypeReference<>(){});
                        String start = DateTimeUtil.formatDateFromUnix(Long.parseLong(gap[1])*1000, null);
                        String end = DateTimeUtil.formatDateFromUnix(Long.parseLong(gap[2])*1000, null);
                        log.info("Loaded {} {} candles between {} and {}", candleList.size(), gap[0], start, end);
                        Collections.reverse(candleList);
                        return candleList;
                    case 429000: //Too Many Requests
                        log.info("Requests are too frequent");
                        Thread.sleep(5000l);
                        return this.candleLoader(gap, retry - 1);
                }
            }
            return Lists.newArrayList();
        } catch (Exception e) {
            log.error(e.getMessage(), e);
            return this.candleLoader(gap, retry - 1);
        }
    }

    private List<String[]> analyticalCandleCaps(String symbol, List<String[]> candleList) {
        long previous = 0l;
        List<String[]> gapList = new ArrayList<>();
        for (String[] candle : candleList) {
            long should = previous + candlestick.getSeconds();
            long openTime = Long.parseLong(candle[0]);
            if(previous > 0 && openTime > should) {
                gapList.add(new String[]{symbol, String.valueOf(should), String.valueOf(openTime)});
            }
            previous = openTime;
        }
        return gapList;
    }

    private File generateFile(CurrencyPairDTO currencyPair) {
        File folder = new File(OUT_FOLDER);
        String fileName = String.format(OUT_FILE, currencyPair.getBaseCurrency(), currencyPair.getQuoteCurrency());
        File file = new File(folder + File.separator + fileName.toLowerCase());
        return file;
    }

    private void generateDataFile(List<String[]> candleList, File datafile) {
        FileOutputStream out = null;
        OutputStreamWriter outWriter = null;
        BufferedWriter bufWrite = null;
        try {
            if(candleList.isEmpty()) {
                log.info("No file output");
                return;
            }
            if (!datafile.exists()) {
                datafile.createNewFile();
            }
            out = new FileOutputStream(datafile);
            outWriter = new OutputStreamWriter(out, "UTF-8");
            bufWrite = new BufferedWriter(outWriter);
            for(String[] candle : candleList) {
                String format =
                                Long.parseLong(candle[0]) + "\t"
                                + candle[1] + "\t"
                                + candle[2] + "\t"
                                + candle[3] + "\t"
                                + candle[4] + "\t"
                                + candle[5] + "\t"
                                + candle[6] + "\n";

                //System.out.println(format);
                bufWrite.write(format);
            }
            log.info("File {} generated successfully", datafile.getName());
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        } finally {
            try {
                if(bufWrite != null )
                    bufWrite.close();
                if(outWriter != null)
                    outWriter.close();
                if(out != null)
                    out.close();
            }catch (IOException e1) {
                log.error(e1.getMessage(), e1);
            }
        }
    }

    private RestTemplate getRestTemplate(String host, Integer port) {
        RestTemplate restTemplate = new RestTemplate();
        if(host != null && port != null) {
            SimpleClientHttpRequestFactory simpleClientHttpRequestFactory = new SimpleClientHttpRequestFactory();
            simpleClientHttpRequestFactory.setProxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(host, port)));
            simpleClientHttpRequestFactory.setConnectTimeout(5000);
            simpleClientHttpRequestFactory.setReadTimeout(5000);
            restTemplate.setRequestFactory(simpleClientHttpRequestFactory);
        }
        return restTemplate;
    }

    enum Candlestick {
        m1("1min",TimeUnit.MINUTES.toSeconds(1L)),
        m3("3min", TimeUnit.MINUTES.toSeconds(3L)),
        m5("5min", TimeUnit.MINUTES.toSeconds(5L)),
        m15("15min", TimeUnit.MINUTES.toSeconds(15L)),
        m30("30min", TimeUnit.MINUTES.toSeconds(30L)),
        h1("1hour", TimeUnit.HOURS.toSeconds(1L)),
        h2("2hour", TimeUnit.HOURS.toSeconds(2L)),
        h4("4hour", TimeUnit.HOURS.toSeconds(4L)),
        h6("6hour", TimeUnit.HOURS.toSeconds(6L)),
        h8("8hour", TimeUnit.HOURS.toSeconds(8L)),
        h12("12hour", TimeUnit.HOURS.toSeconds(12L)),
        d1("1day", TimeUnit.DAYS.toSeconds(1L)),
        d3("1week", TimeUnit.DAYS.toSeconds(3L));

        private final String code;
        private final Long seconds;

        Candlestick(String code, Long seconds) {
            this.seconds = seconds;
            this.code = code;
        }
        public Long getSeconds() {
            return this.seconds;
        }
        public String code() {
            return this.code;
        }
    }
}

Test:

public static void main(String[] args) {
    CandleHistoryBackFill candleHistoryBackFill = new CandleHistoryBackFill();
    candleHistoryBackFill.backfillFrom(LocalDate.of(2022, 1, 10).atStartOfDay());
    candleHistoryBackFill.backfillHistory(new CurrencyPairDTO("DOGE", "USDT"));
}

[hlevel]

@hlevel I think you should use p.getOpeningOrder() and get the field you want. Some people think getDescription() already returns too much information.

Hello, you can add a more detailed method Let me show you an example:

[1]2021-12-11 21:47:59.999   LTC         BUY   0.129572074323 @ $156.000000000000 + PENDING     worth $20.213243594529803 USDT
[1]2021-12-11 21:48:59.999   LTC         BUY   0.129572074323 @ $156.000000000000 + FILLED      spent $20.213243594529 USDT duration: 0:01:60. Worth $156.000000000000 USDT (free $0.002043783539 USDT)
[1]2021-12-11 22:34:59.999   LTC         SELL  0.129572074323 @ $155.300000000000 - PENDING     worth $20.12254314250307 USDT. Expected P/L -0.65% | 47 ticks [min 156.700000000000 (-0.55%), max $156.700000000000 (0.35%))] stop loss
[1]2021-12-11 22:35:59.999   LTC         SELL  0.129572074323 @ $155.300000000000 - FILLED      sold $20.122543142503 USDT duration: 0:01:60. P/L USDT -0.131036238764 (-0.65%) Holdings $20.104464382899 USDT (free $20.104464382899)
[2]2021-12-12 00:49:59.999   LTC         BUY   0.127264584806 @ $157.800000000000 + PENDING     worth $20.08235148252505 USDT
[2]2021-12-12 00:50:59.999   LTC         BUY   0.127264584806 @ $157.800000000000 + FILLED      spent $20.082351482525 USDT duration: 0:01:60. Worth $157.800000000000 USDT (free $0.002030548892 USDT)
[2]2021-12-12 01:04:59.999   LTC         SELL  0.127264584806 @ $157.200000000000 - PENDING     worth $20.00599273164092 USDT. Expected P/L -0.58% | 15 ticks [min 157.900000000000 (-0.48%), max $157.900000000000 (-0.04%))] stop loss
[2]2021-12-12 01:05:59.999   LTC         SELL  0.127264584806 @ $157.200000000000 - FILLED      sold $20.005992731640 USDT duration: 0:01:60. P/L USDT -0.116447095099 (-0.58%) Holdings $19.988017287801 USDT (free $19.988017287801)

[straumat]

@hlevel can you provide the method code? We could indeed add a getDetailedDescription() method in PositionDTO. If you feel comfortable with it, you could do a PR

[hlevel]

@hlevel can you provide the method code? We could indeed add a getDetailedDescription() method in PositionDTO. If you feel comfortable with it, you could do a PR

I have no code for this detail. This paragraph is for other projects.